System using image connectivity to reduce bundle size for bundle adjustment

Patent metadata

Field	Value
Publication number	US-11948329-B2
Application number	US-202218057061-A
Country	US
Kind code	B2
Filing date	Nov 18, 2022
Priority date	Oct 25, 2019
Publication date	Apr 2, 2024
Grant date	Apr 2, 2024

Abstract

Systems and methods are disclosed, including a non-transitory computer readable medium storing computer executable instructions that when executed by a processor cause the processor to identify a first image, a second image, and a third image, the first image overlapping the second image and the third image, the second image overlapping the third image; determine a first connectivity between the first image and the second image; determine a second connectivity between the first image and the third image; determine a third connectivity between the second image and the third image, the second connectivity being less than the first connectivity, the third connectivity being greater than the second connectivity; assign the first image, the second image, and the third image to a cluster based on the first connectivity and the third connectivity; conduct a bundle adjustment process on the cluster of the first image, the second image, and the third image.

First claim

What is claimed is:

1. A non-transitory computer readable medium storing computer executable instructions that when executed by one or more processors cause the one or more processors to:
assign overlapping images depicting a geographic region into a cluster based on a connectivity of pairs of overlapping images, wherein connectivity is an amount of overlap between two images in the overlapping images, wherein overlap is based on one or more of: an amount of geographic area depicted in both of the two images, and, a number of shared feature points depicted in both of the two images;
form a match graph comprising nodes indicative of the images in the cluster and edges connecting the nodes, the edges indicative of the connectivity between the images in the cluster;
determine the cluster is above a predetermined bundle adjustment threshold;
utilize a graph-cut algorithm on the match graph to bi-section the cluster into a first sub-cluster and a second sub-cluster, the first sub-cluster sharing at least one node with the second sub-cluster, the first sub-cluster having at least one node not in the second sub-cluster, the second sub-cluster having at least one node not in the first sub-cluster; and
conduct a bundle adjustment process on the images within at least one of the first sub-cluster and the second sub-cluster.

2. The non-transitory computer readable medium of claim 1 , wherein the bundle adjustment process comprises:
building feature tracks of features from the images in the sub-cluster;
reducing a number of feature tracks in the sub-cluster; and
bundling the feature tracks.

3. The non-transitory computer readable medium of claim 2 , wherein reducing the number of feature tracks in the sub-cluster comprises removing feature tracks having a track quality metric that is below a track quality threshold.

4. The non-transitory computer readable medium of claim 3 , wherein the track quality metric is based on one or more of: a feature track triangularization quality, a feature point quantity within the feature track, a mean reprojection error for the feature track, and a quantity of feature tracks in the sub-cluster.

5. The non-transitory computer readable medium of claim 2 , wherein reducing the number of feature tracks in the sub-cluster comprises removing feature tracks utilizing a spatial filter.

6. The non-transitory computer readable medium of claim 2 , wherein reducing the number of feature tracks in the sub-cluster comprises bi-sectioning the sub-cluster into smaller sub-clusters having no connectivity between the smaller sub-clusters.

7. The non-transitory computer readable medium of claim 1 , wherein the overlapping images are associated with camera pose data of one or more cameras that captured the overlapping images, and wherein conducting the bundle adjustment process on the images within the sub-cluster produces adjusted camera pose data, the non-transitory computer readable medium storing computer executable instructions that when executed by one or more processors cause the one or more processors to:
triangulate one or more of the images utilizing the adjusted camera pose data.

8. The non-transitory computer readable medium of claim 1 , wherein the sub-clusters include a first sub-cluster and a second sub-cluster sharing at least one image, the non-transitory computer readable medium storing computer executable instructions that when executed by one or more processors cause the one or more processors to:
align the first sub-cluster with the second sub-cluster into the cluster, based on the at least one shared image, after conducting the bundle adjustment process.

9. The non-transitory computer readable medium of claim 6 , wherein bi-sectioning the sub-cluster into smaller sub-clusters includes utilizing the match graph.

10. The non-transitory computer readable medium of claim 1 , wherein the predetermined bundle adjustment threshold is based on one or more of: a quantity of the images in the cluster, a quantity of feature points in the cluster, the connectivity between the pairs of overlapping images, feature point quality, and feature match quality.

11. A method, comprising:
assigning, by one or more computer processors, overlapping images depicting a geographic region into a cluster based on a connectivity of pairs of overlapping images, wherein connectivity is an amount of overlap between two images in the overlapping images, wherein overlap is based on one or more of: an amount of geographic area depicted in both of the two images, and, a number of shared feature points depicted in both of the two images;
forming, by the one or more computer processors, a match graph comprising nodes indicative of the images in the cluster and edges connecting the nodes, the edges indicative of the connectivity between the images in the cluster;
determining, by the one or more computer processors, the cluster is above a predetermined bundle adjustment threshold;
utilizing, by the one or more computer processors, a graph-cut algorithm on the match graph to bi-section the cluster into a first sub-cluster and a second sub-cluster, the first sub-cluster sharing at least one node with the second sub-cluster, the first sub-cluster having at least one node not in the second sub-cluster, the second sub-cluster having at least one node not in the first sub-cluster; and
conducting, by the one or more computer processors, a bundle adjustment process on the images within at least one of the first sub-cluster and the second sub-cluster.

12. The method of claim 11 , wherein the bundle adjustment process comprises:
building feature tracks of features from the images in the sub-cluster;
reducing a number of feature tracks in the sub-cluster; and
bundling the feature tracks.

13. The method of claim 12 , wherein reducing the number of feature tracks in the sub-cluster comprises removing feature tracks having a track quality metric that is below a track quality threshold.

14. The method of claim 13 , wherein the track quality metric is based on one or more of: a feature track triangularization quality, a feature point quantity within the feature track, a mean reprojection error for the feature track, and a quantity of feature tracks in the sub-cluster.

15. The method of claim 12 , wherein reducing the number of feature tracks in the sub-cluster comprises removing feature tracks utilizing a spatial filter.

16. The method of claim 12 , wherein reducing the number of feature tracks in the sub-cluster comprises bi-sectioning the sub-cluster into smaller sub-clusters having no connectivity between the smaller sub-clusters.

17. The method of claim 11 , wherein the overlapping images are associated with camera pose data of one or more cameras that captured the overlapping images, and wherein conducting the bundle adjustment process on the images within the sub-cluster produces adjusted camera pose data, the method further comprising:
triangulating, by the one or more computer processors, one or more of the images utilizing the adjusted camera pose data.

18. The method of claim 11 , wherein the sub-clusters include a first sub-cluster and a second sub-cluster sharing at least one image, the method further comprising:
aligning the first sub-cluster with the second sub-cluster into the cluster, based on the at least one shared image, after conducting the bundle adjustment process.

19. The method of claim 16 , wherein bi-sectioning the sub-cluster into smaller sub-clusters includes utilizing the match graph.

20. The method of

Assignees

• Pictometry Int Corp

Inventors

• Nilosek David

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/457

  by analysing connectivity, e.g. edge linking, connected component analysis or slices · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title