3D surface estimation and prediction to boost fidelity of realtime LiDAR model generation

Patent metadata

Field	Value
Publication number	US-10979644-B2
Application number	US-201916268696-A
Country	US
Kind code	B2
Filing date	Feb 6, 2019
Priority date	Feb 6, 2019
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

Systems and methods for capturing images of a target area that include controlling an image sensor mounted to a gimbal to capture images of the target area over time and controlling the gimbal using an elevator algorithm to adjust the orientation of the image sensor.

First claim

What is claimed is:

1. A computer implemented method for image capture of a target area, comprising:
controlling a single image sensor mounted to a gimbal to capture images of the target area over time;
moving the gimbal along a single axis using an elevator algorithm to scan the target area with the single image sensor; and
controlling the gimbal using the elevator algorithm to adjust the orientation of the single image sensor along at least two degrees of freedom to build a 3D image model of the target area using the single image sensor.

2. The computer implemented method of claim 1 , wherein controlling the gimbal comprises using the elevator algorithm to move the gimbal to adjust the orientation of the image sensor along three degrees of freedom to build a 3D image model of the target area over time from 3D angular positions of the image sensor.

3. The computer implemented method of claim 2 , wherein controlling the gimbal comprises moving of the gimbal to obtain images of a sub-area within the target area.

4. The computer implemented method of claim 2 , wherein controlling the gimbal comprises moving of the gimbal to obtain images of an object moving within the target area thereby capturing images of the object as the object moves within the target area over time.

5. The computer implemented method of claim 2 , further including training a Deep Neural Network (DNN) on the captured images to increase a fidelity of the 3D image model of the target area.

6. The computer implemented method of claim 5 , further including controlling the movement of the gimbal based on a confidence level of the DNN increased fidelity 3D image model, whereby image capture of areas of lower confidence level are prioritized higher than areas of higher confidence level.

7. The computer implemented method of claim 2 , wherein using the elevator algorithm includes reducing the 3D angular positions of the image sensor to a single-dimensional proxy using a Z-order curve.

8. The computer implemented method of claim 1 , wherein the method is provided as a service in a cloud environment.

9. A system for image capture of a target area, comprising:
one or more storage devices;
one or more hardware processors coupled to the one or more storage devices;
one or more hardware processors operable to control a single image sensor mounted to a gimbal to capture images of the target area over time;
one or more hardware processors operable to move the gimbal along a single axis using an elevator algorithm to scan the target area with the single image sensor; and
one or more hardware processors operable to control the gimbal using the elevator algorithm to adjust the orientation of the single image sensor along at least two degrees of freedom to build a 3D image model of the target area using the single image sensor.

10. The system of claim 9 , wherein the system further comprises one or more hardware processors operable to control the gimbal comprises using the elevator algorithm to move the gimbal to adjust the orientation of the image sensor along three degrees of freedom to build a 3D image model of the target area over time from 3D angular positions of the image sensor.

11. The system of claim 10 , wherein the one or more hardware processors are further configured to control the gimbal by moving of the gimbal to obtain images of a sub-area within the target area.

12. The system of claim 10 , wherein the one or more hardware processors are further configured to control the gimbal by moving of the gimbal to obtain images of an object moving within the target area thereby capturing images of the object as the object moves within the target area over time.

13. The system of claim 10 , wherein the one or more hardware processors are further configured to training a Deep Neural Network (DNN) on the captured images to increase a fidelity of the 3D image model of the target area.

14. The system of claim 13 , wherein the one or more hardware processors are further configured to control the movement of the gimbal based on a confidence level of the DNN increased fidelity 3D image model, whereby image capture of areas of lower confidence level are prioritized higher than areas of higher confidence level.

15. The system of claim 10 , wherein the one or more hardware processors are further configured to use the elevator algorithm to reduce the 3D angular positions of the image sensor to a single-dimensional proxy using a Z-order curve.

16. A computer readable storage medium storing a program of instructions executable by a machine to perform a method of capturing images of a target area, the method comprising:
controlling a single image sensor mounted to a gimbal to capture images of the target area over time;
moving the gimbal along a single axis using an elevator algorithm to scan the target area with the single image sensor; and
controlling the gimbal using the elevator algorithm to adjust the orientation of the single image sensor along at least two degrees of freedom to build a 3D image model of the target area using the single image sensor.

17. The computer readable storage medium of claim 16 , wherein the method further comprises the steps of:
controlling the gimbal comprising using the elevator algorithm to move the gimbal to adjust the orientation of the image sensor along three degrees of freedom to build a 3D image model of the target area over time from 3D angular positions of the image sensor.

18. The computer readable storage medium of claim 17 , wherein the method further comprises the steps of:
controlling the gimbal comprising moving of the gimbal to obtain images of a sub-area within the target area.

19. The computer readable storage medium of claim 17 , wherein the method further comprises the steps of:
controlling the gimbal comprising moving of the gimbal to obtain images of an object moving within the target area thereby capturing images of the object as the object moves within the target area over time.

20. The computer readable storage medium of claim 17 , wherein using the elevator algorithm includes reducing the 3D angular positions of the image sensor to a single-dimensional proxy using a Z-order curve.

21. A computer implemented method for image capture of a target area, comprising:
controlling an image sensor mounted to a gimbal to capture images of the target area over time;
controlling the gimbal using an elevator algorithm to adjust the orientation of the image sensor, including using the elevator algorithm to move the gimbal to adjust the orientation of the image sensor along three degrees of freedom to build a 3D image model of the target area over time from 3D angular positions of the image sensor;
training a Deep Neural Network (DNN) on the captured images to increase a fidelity of the 3D image model of the target area; and
controlling the movement of the gimbal based on a confidence level of the DNN increased fidelity 3D image model, whereby image capture of areas of lower confidence level are prioritized higher for recapture than areas of higher confidence level.

22. The computer implemented method of claim 21 , further comprising training the DNN to predict a dense 3D point cloud or mesh.

23. The computer implemented method of claim 22 , further comprising constructing the prediction by artificially downsampling a high-resolution point cloud or mesh.

24. A computer implemented method for image capture of a target area, comprising:
c

Assignees

• IBM

Inventors

• Jamjoom Hani T
• Manweiler Justin G

Classifications

• H04N23/695 Primary

  Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects · CPC title

• H04N23/50

  Constructional details · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title