Submersible inspection device and vision based modelling

Patent metadata

Field	Value
Publication number	US-11468628-B2
Application number	US-201715835178-A
Country	US
Kind code	B2
Filing date	Dec 7, 2017
Priority date	Dec 7, 2016
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

A submersible vehicle which includes a plurality of cameras can be used to collect visual images of an object of interest submerged in a liquid environment, such as in a tank (e.g. transformer tank). In one form the submersible vehicle is remotely operated such as an ROV or an autonomous vehicle. Image information from the submersible along with inertial measurements in some embodiments is used with a vision based modelling system to form a model of an internal object of interest in the tank. The vision based modelling system can include a number of processes to form the model such as but not limited to tracking, sparse and dense reconstruction, model generation, and rectification.

First claim

What is claimed is:

1. A method performed by a processing device, the method comprising:
viewing an object with a plurality of cameras on a submersible immersed in a liquid to form a first series of images;
estimating a pose of the object in the first series of images;
performing a bundle adjustment on features of the object in the first series of images;
computing depth-maps based on the first series of images and the bundle adjustment to form a 3D dense reconstruction;
creating a point cloud based upon fusing individual depth-maps from the depth-maps of the 3D dense reconstruction;
converting the point cloud to a mesh to form a model;
rectifying the model with pre-existing data of the object, the pre-existing data of the object including a prior model of the object from a prior inspection by the submersible that included prior viewing of the object to form a second series of images, estimating of the pose in the second series of images, performing of the bundle adjustment, computing of the depth-maps, creating of the point cloud, and converting of the point cloud to the mesh to form the prior model of the object from the prior inspection; and
outputting the model for display.

2. The method of claim 1 , wherein the viewing the object is performed beneath a surface of the liquid.

3. The method of claim 1 , which further includes recording a pose orientation of the submersible along with images taken at the pose orientation.

4. The method of claim 1 , wherein the rectifying of the model with pre-existing data of the model is performed with a CAD model.

5. The method of claim 1 , which further includes introducing at least one of a texture and an annotation to the model.

6. The method of claim 1 , wherein the bundle adjustment is performed in each camera to generate sparse maps from each camera.

7. The method of claim 1 , wherein the converting also includes projecting a local neighborhood of a point along the point's normal, and connecting unconnected points.

8. The method of claim 1 , wherein the converting is performed without telemetry.

9. An apparatus comprising:
a vision based modelling system for generating a model of a submerged object of interest located in a working liquid, the vision based modelling system structured to:
capture a set of images from a plurality of cameras mounted on a submersible vehicle, the set of images comprising a first plurality of images;
estimate a pose of an object in the set of images;
perform a bundle adjustment on features of the object in the set of images;
create a point cloud upon fusing individual depth-maps based from the set of images and the bundle adjustment;
convert the point cloud to a mesh to form a model;
rectify the model with pre-existing data of the object, the pre-existing data comprising a prior model of the object from a previous inspection that produced the prior model using the vision based modelling system that includes the capture of a set of images comprising a second plurality of images, estimate of the pose, perform of the bundle adjustment, create of the point cloud, and convert of the point cloud to the mesh to form the prior model of the object; and
output the model for display.

10. The apparatus of claim 9 , wherein the vision based modelling system structured to compute depth-maps based on the set of images to form a 3D dense reconstruction.

11. The apparatus of claim 9 , which further includes a computer having a non-transitory computer readable memory, the vision based modelling system expressed as a programming instruction and stored in the non-transitory computer readable memory.

12. The apparatus of claim 9 , wherein the vision based modelling system is hosted in a distributed computing environment having at least two computers.

13. The apparatus of claim 9 , wherein the vision based modelling system is further structured to introduce at least one of a texture and an annotation to the model, and wherein the bundle adjustment is performed on images from each camera to generate sparse maps of the images from each camera.

14. The apparatus of claim 9 , wherein the vision based modelling system is further structured to project a local neighborhood of a point along the point's normal, and connecting unconnected points; and which further includes a submersible vehicle having a plurality of cameras, and wherein the vision based modelling system is further structured to store a pose orientation of the submersible vehicle with image frames taken at the pose orientation.

15. An apparatus comprising:
a first computer comprising a processing device and a memory, the first computer structured to receive a first plurality of images of an object as viewed through a liquid from a plurality of cameras on board a submersible vehicle and to execute a vision based modelling system; and
the vision based modelling system configured to execute instructions to: determine a pose estimate of the object, utilize bundle adjustment, provide a point cloud, convert the point cloud to a mesh to form a vision based model, and utilize stored information about the object to rectify the vision based model with the stored information about the object, the stored information comprising a prior model created using the first computer to receive a second plurality of images from the plurality of cameras and execute the vision based modelling system to determine the pose estimate, utilize the bundle adjustment, provide the point cloud, and convert the point cloud to the mesh to form the prior model, a vision based model created from the model generation module, and
the first computer further configured to execute at least one of the 2D tracker module, 3D sparse reconstruction module, 3D dense reconstruction module, model generation module, and image rectification module.

16. The apparatus of claim 15 , wherein the liquid is comprised in a tank, the apparatus further comprising a submersible vehicle that includes the plurality of cameras.

17. The apparatus of claim 15 , wherein the instructions that utilize bundle adjustment perform a global bundle adjustment with telemetry integration.

18. The apparatus of claim 15 , wherein the instructions that provide the point cloud further determine depth-maps using information from the bundle adjustment.

19. The apparatus of claim 15 , wherein the prior model is a CAD model or a prior vision based model formed from the vision based modelling system.

Assignees

• Hitachi Energy Switzerland Ag

Inventors

• Cole Gregory
• Eakins William
• Lasko Daniel
• Shah Harshang
• Fuhlbrigge Thomas
• Morato Carlos W
• Zhang Biao
• Cheim Luiz
• Patel Poorvi
• Staab Harald Josef
• Sharma Saumya

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T7/593

  from stereo images · CPC title

• B63G8/001

  Underwater vessels adapted for special purposes, e.g. unmanned underwater vessels; Equipment specially adapted therefor, e.g. docking stations (self-propelled or direction controlled diving chambers with mechanical link to a base B63C11/42) · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• B63G2008/005

  remotely controlled · CPC title