High-quality stereo reconstruction featuring depth map alignment and outlier identification

Patent metadata

Field	Value
Publication number	US-9373190-B2
Application number	US-201414327094-A
Country	US
Kind code	B2
Filing date	Jul 9, 2014
Priority date	Jul 9, 2014
Publication date	Jun 21, 2016
Grant date	Jun 21, 2016

Abstract

A novel stereo reconstruction pipeline that features depth map alignment and outlier identification is provided. One example method includes obtaining a plurality of images depicting a scene. The method includes determining a pose for each of the plurality of images. The method includes determining a depth map for each of the plurality of images such that a plurality of depth maps are determined. Each of the plurality of depth maps describes a plurality of points in three-dimensional space that correspond to objects in the scene. The method includes aligning the plurality of depth maps by transforming one or more of the plurality of depth maps so as to improve an alignment between the plurality of depth maps. The method includes identifying one or more outlying points. The method includes generating a three-dimensional model of the scene based at least in part on the plurality of depth maps.

First claim

What is claimed is:

1. A computer-implemented method for generating three-dimensional models, the method comprising:
obtaining, by one or more computing devices, a plurality of images depicting a scene;
determining, by the one or more computing devices, a pose for each of the plurality of images;
determining, by the one or more computing devices, a depth map for each of the plurality of images such that a plurality of depth maps are determined, wherein each of the plurality of depth maps describes a plurality of points in three-dimensional space that correspond to objects in the scene;
aligning, by the one or more computing devices, the plurality of depth maps by transforming one or more of the plurality of depth maps so as to improve an alignment between the plurality of depth maps;
after aligning the plurality of depth maps, identifying, by the one or more computing devices, one or more of the plurality of points described by one or more of the plurality of depth maps as one or more outlying points; and
generating, by the one or more computing devices, a three-dimensional model of the scene based at least in part on the plurality of aligned depth maps.

2. The computer-implemented method of claim 1 , wherein identifying, by the one or more computing devices, one or more of the plurality of points described by one or more of the plurality of depth maps as one or more outlying points comprises:
determining, by the one or more computing devices for each point described by one of the plurality of depth maps, a number of points described by other depth maps that are within a threshold distance from such point; and
determining, by the one or more computing devices for each point described by one of the plurality of depth maps, whether such point is an outlying point based at least in part on the number of points described by other depth maps that are within a threshold distance from such point.

3. The computer-implemented method of claim 2 , wherein determining, by the one or more computing devices for each point described by one of the plurality of depth maps, whether such point is an outlier based at least in part on the number of points described by other depth maps that are within a threshold distance from such point comprises:
determining, by the one or more computing devices for each point described by one of the plurality of depth maps, a number of other depth maps represented by the number of points described by other depth maps that are within a threshold distance from such point; and
determining, by the one or more computing devices for each point described by one of the plurality of depth maps, whether such point is an outlier based at least in part on the number of other depth maps represented by the number of points described by other depth maps that are within a threshold distance from such point.

4. The computer-implemented method of claim 1 , further comprising reducing one or more confidence scores respectively associated with the one or more outlying points, wherein the confidence scores are used as weightings for a volumetric fusion technique performed to merge the plurality of depth maps.

5. The computer-implemented method of claim 1 , wherein aligning, by the one or more computing devices, the plurality of depth maps by transforming one or more of the plurality of depth maps so as to improve the alignment between the plurality of depth maps comprises transforming, by the one or more computing devices, one or more of the plurality of depth maps so as to minimize an objective function describing the alignment between the plurality of depth maps.

6. The computer-implemented method of claim 5 , wherein:
the objective function comprises a distance term;
the plurality of depth maps comprise a plurality of pairs of depth maps, wherein each pair of depth maps comprises a source depth map and a target depth map, and wherein the source depth map and the target depth map for each pair of depth maps exhibit an overlap with respect to one another; and
the distance term comprises a sum, for all of the plurality of pairs of depth maps, of a plurality of squared distances respectively between one or more of the plurality of points described by the source depth map of each pair of depth maps and one or more planes respectively associated with one or more of the plurality of points described by the target depth map of such pair of depth maps.

7. The computer-implemented method of claim 6 , wherein the distance term allows for transformation of both the source depth map and the target depth map for each pair of depth maps.

8. The computer-implemented method of claim 7 , wherein
each of the plurality of depth maps describe their respective plurality of points in three-dimensional space according to a local coordinate system; and
the source depth map and the target depth map are both transformed in the distance term relative to their respective local coordinate systems.

9. The computer-implemented method of claim 6 , wherein:
the one or more of the plurality of points described by the source depth map comprise a random subset of the plurality of points described by the source depth map; and
the one or more planes respectively associated with the one or more of the plurality of points described by the target depth map comprise planes respectively associated with a plurality of closest points of the source depth map that are respectively closest to the random subset of the plurality of points described by the source depth map.

10. The computer-implemented method of claim 5 , wherein:
the objective function comprises a distance term; and
the distance term comprises a sum of a plurality of squared point-to-plane distances respectively associated with at least one of the plurality of points from each of the plurality of depth maps and a plane associated with at least one of the plurality of points for at least one of the other depth maps.

11. The computer-implemented method of claim 5 , wherein transforming, by the one or more computing devices, one or more of the plurality of depth maps so as to minimize the objective function describing the alignment between the plurality of depth maps comprises iteratively minimizing, by the one or more computing devices, the objective function, wherein the objective function allows for transformation of all of the plurality of depth maps, such that each of the plurality of depth maps is iteratively transformed to improve the alignment.

12. The computer-implemented method of claim 5 , wherein:
the objective function comprises a regularization term; and
the regularization term describes an amount of transformation performed for each of the plurality of depth maps.

13. The computer-implemented method of claim 1 , wherein generating, by the one or more computing devices, the three-dimensional model of the scene based at least in part on the plurality of depth maps comprises:
merging, by the one or more computing devices, the plurality of depth maps to create a signed distance function, wherein the merging comprises determining, for each of a plurality of voxels included in a volume, a weighted average of a plurality of voxel-to-surface distances respectively provided by the plurality of depth maps, and wherein a weighting provided for each depth map is determined based at least in part on whether one of the outlying points was used to determine the voxel-to-surface distance provided by such depth map; and
generating, by the one or more computing devices, a mesh model of the scene based at least in part on the signed distance function.

14. One or more non-transitory computer-readable media s

Assignees

• Google Inc

Inventors

• Wei Xiaolin
• Golovinskiy Aleksey

Classifications

• G06T17/05 Primary

  Geographic models · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/30184

  Infrastructure · CPC title

• G06T7/33

  using feature-based methods · CPC title