Real-time camera position estimation with drift mitigation in incremental structure from motion

Patent metadata

Field	Value
Publication number	US-10269147-B2
Application number	US-201715582987-A
Country	US
Kind code	B2
Filing date	May 1, 2017
Priority date	May 1, 2017
Publication date	Apr 23, 2019
Grant date	Apr 23, 2019

Abstract

A system provides camera position and point cloud estimation 3D reconstruction. The system receives images and attempts existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption. If existing structure integration fails, the system attempts dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction. If overlaps are found, the system matches the images with the overlaps to determine a highest probability frame from the overlaps, and attempts existing structure integration again. If overlaps are not found or existing structure integration fails again, the system attempts bootstrapping based on the images. If any of existing structure integration, dictionary overlap detection, or bootstrapping succeeds, and if multiple disparate tracks have come to exist, the system attempts reconstructed track merging.

First claim

What is claimed is:

1. A method for camera position and point cloud estimation for three-dimensional (3D) reconstruction, comprising:
receiving images;
attempting a first existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption;
if the first existing structure integration fails, attempting a dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction;
if overlaps are found, matching the images with the overlaps to determine a highest probability frame from the overlaps and attempting a second existing structure integration under the sequential image reception assumption;
if overlaps are not found or the second existing structure integration fails, attempting bootstrapping based on the images; and
if one of the first existing structure integration, the dictionary overlap detection, the second existing structure integration, or the bootstrapping succeeds, and if multiple disparate tracks have come to exist, attempting reconstructed track merging.

2. The method of claim 1 , wherein the images comprise a new image and corresponding matched images, wherein the first existing structure integration comprises:
using feature matches to find existing world points in the new image under the sequential image reception assumption;
if sufficient existing world points are found, estimating an initial relative pose;
refining the initial relative pose to obtain a refined pose;
nominating additional existing overlap frames through spatial partitioning;
matching the new image with the additional existing overlap frames;
making a copy of the refined pose for each overlap frame and refining each copy;
discarding all pose copies and combining all inliers from each overlap frame;
refining the refined pose using combined inliers from the additional existing overlap frames to obtain a further refined pose and add it to the spatial partitioning;
triangulating all new inlier points and adding them to a structure database;
performing bundle adjustment over overlap poses and points with most views;
triangulating all affected world points; and
updating active point status across all affected points and views.

3. The method of claim 2 , wherein the copies of the refined pose are refined in parallel threads.

4. The method of claim 2 , wherein the refining of each copy of the refined pose is performed using only matches for a corresponding overlap frame.

5. The method of claim 2 , wherein the bundle adjustment is run on a network of triangulated points and poses for each added frame by observing poses not included in overlapping frames and locking world points in the overlapping frames.

6. The method of claim 2 , wherein the refining of the initial relative pose comprises applying iterative pose improvement on the initial relative pose.

7. The method of claim 2 , wherein the refining of the initial relative pose comprises inlier tightening, a sequence of parallelized triangulations, and non-linear refinement with variable error margins, to obtain the refined pose.

8. The method of claim 2 , wherein each point is flagged as active if it has sufficient observers to be considered stable and it does not have a degenerate angle of incidence.

9. The method of claim 2 , wherein all structure changes are enqueued for visualization.

10. The method of claim 1 , wherein estimated structures are integrated into a structure database.

11. The method of claim 10 , wherein the structure database comprises a global relational data structure including all poses, world points, images, and their relationships, and including a caching mechanism and data structures for relating data.

12. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to provide camera position and point cloud estimation for three-dimensional (3D) reconstruction, the processor:
receiving images;
attempting a first existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption;
if the first existing structure integration fails, attempting a dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction;
if overlaps are found, matching the images with the overlaps to determine a highest probability frame from the overlaps and attempting a second existing structure integration under the sequential image reception assumption;
if overlaps are not found or the second existing structure integration fails, attempting bootstrapping based on the images; and
if one of the first existing structure integration, the dictionary overlap detection, the second existing structure integration, or the bootstrapping succeeds, and if multiple disparate tracks have come to exist, attempting reconstructed track merging.

13. The computer readable medium of claim 12 , wherein the images comprise a new image and corresponding matched images, wherein the first existing structure integration comprises:
using feature matches to find existing world points in the new image under the sequential image reception assumption;
if sufficient existing world points are found, estimating an initial relative pose;
refining the initial relative pose to obtain a refined pose;
nominating additional existing overlap frames through spatial partitioning;
matching the new image with the additional existing overlap frames;
making a copy of the refined pose for each overlap frame and refining each copy;
discarding all pose copies and combining all inliers from each overlap frame;
refining the refined pose using combined inliers from the additional existing overlap frames to obtain a further refined pose and add it to the spatial partitioning;
triangulating all new inlier points and adding them to a structure database;
performing bundle adjustment over overlap poses and points with most views;
triangulating all affected world points; and
updating active point status across all affected points and views.

14. The computer readable medium of claim 13 , wherein the copies of the refined pose are refined in parallel threads.

15. The computer readable medium of claim 13 , wherein the refining of each copy of the refined pose is performed using only matches for a corresponding overlap frame.

16. The computer readable medium of claim 13 , wherein the bundle adjustment is run on a network of triangulated points and poses for each added frame by observing poses not included in overlapping frames and locking world points in the overlapping frames.

17. The computer readable medium of claim 13 , wherein the refining of the initial relative pose comprises applying iterative pose improvement on the initial relative pose.

18. The computer readable medium of claim 13 , wherein the refining of the initial relative pose comprises inlier tightening, a sequence of parallelized triangulations, and non-linear refinement with variable error margins, to obtain the refined pose.

19. The computer readable medium of claim 13 , wherein each point is flagged as active if it has sufficient observers to be considered stable and it does not have a degenerate angle of incidence.

20. A system for camera position and point cloud estimation for three-dimensional (3D) reconstruction, the system comprising:
a processor; and
a memory coupled with the processor an

Assignees

• Lockheed Corp

Inventors

• Jones Michael
• Dickin Adam James

Classifications

• G06V20/17

  taken from planes or by drones · CPC title

• G06V20/13 Primary

  Satellite images · CPC title

• G06T12/10

  Image preprocessing, e.g. calibration, positioning of sources or scatter correction · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V10/757

  Matching configurations of points or features · CPC title