Capturing and aligning three-dimensional scenes

Patent metadata

Field	Value
Publication number	US-10909770-B2
Application number	US-201314070430-A
Country	US
Kind code	B2
Filing date	Nov 1, 2013
Priority date	Feb 24, 2012
Publication date	Feb 2, 2021
Grant date	Feb 2, 2021

Abstract

Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor.

First claim

What is claimed is:

1. A method comprising:
determining, by a system comprising a processor, a global alignment, relative to a three-dimensional coordinate space, between three or more sets of three-dimensional data captured from an object or an environment at different capture positions or orientations, wherein the three or more sets of three-dimensional data respectively correspond to volumes of the object or the environment and wherein at least some of the volumes overlap;
identifying, by the system, a first set of a second set of three-dimensional data from the three or more sets of three-dimensional data that correspond to non-overlapping volumes of the object or the environment;
determining, by the system, a first set of visual features included in a first two-dimensional image associated with the first set of three-dimensional data;
determining, by the system, a second set of visual features included in a second two-dimensional image associated with the second set of three-dimensional data; and
determining, by the system, an alignment score representative of a quality of a pair-wise alignment between the first set of three-dimensional data and the second set of three-dimensional data based on a comparison of the first set of visual features and the second set of visual features.

2. The method of claim 1 , further comprising:
employing, by the system, the alignment score to generate an optimized pair-wise alignment between the first set of three-dimensional data and the second set of three-dimensional data, thereby reducing a degree of drift associated with the global alignment.

3. The method of claim 1 , further comprising:
identifying, by the system, pairs of features respectively comprising a first visual feature included in the first set of visual features that corresponds to a second visual feature included in the second set of visual features; and
determining, by the system, spatial distances associated with the pairs of features, comprising:
for each pair of the pairs of features, determining a distance between the first visual feature and the second visual feature based on a defined spatial position of the second visual feature relative to the three-dimensional coordinate space; and

determining, by the system, the alignment score based on the spatial distances.

4. The method of claim 3 , wherein the identifying the pairs of features comprises identifying the pairs of features based on a visual correspondence requirement associated with visual correspondence between the first visual feature and the second visual feature.

5. The method of claim 3 , further comprising:
generating, by the system, a visual representation of the object or environment based on the second set of three-dimensional data from a perspective of a virtual camera at a position and orientation from which the first set of three-dimensional data was captured, and
wherein the determining the spatial distances comprises:
for each pair of the pairs of features, determining the distance between the first visual feature as included in the first two-dimensional image and the second visual feature as included in the visual representation.

6. The method of claim 3 , further comprising, based on the alignment score being less than an optimal score:
determining, by the system, a modification to a position or orientation of the first set of three-dimensional data relative to the second set of three-dimensional data that facilitates minimizing the spatial distances; and
implementing, by the system, the modification to adjust the pair-wise alignment, thereby resulting in an optimized pair-wise alignment.

7. The method of claim 6 , further comprising:
adjusting, by the system, the global alignment based on the optimized pair-wise alignment, thereby resulting in an optimized.

8. The method of claim 3 , wherein the identifying the pairs of features comprises:
defining, by the system, a volumetric region along a ray extending from a position and orientation at which the first set of three-dimensional data was captured through the first two-dimensional image; and
identifying, by the system, the pairs of features based on inclusion of the second feature of the pairs of features within the volumetric region.

9. The method of claim 3 , wherein the identifying the pairs of features comprises:
projecting, by the system, the second set of visual features in a three-dimensional space from a perspective of a virtual camera at a position and orientation from which the first set of three-dimensional data was captured,
wherein the determining the spatial distances comprises:
for each pair of the pairs of features, determining the distance between the first visual feature as included in the first two-dimensional image and the second visual feature as projected in the three-dimensional space.

10. The method of claim 3 , further comprising:
generating, by the system, a visual representation of the object or environment based on the second set of three-dimensional data from a perspective of a virtual camera at a position and orientation from which the first set of three-dimensional data was captured, and
wherein the determining the spatial distances comprises:
for each pair of the pairs of features, determining visual similarity between the first visual feature as included in the first two-dimensional image and the second visual feature as included in the visual representation.

11. A system comprising,
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
determining an alignment, relative to a three-dimensional coordinate space, between first three-dimensional data captured from an object or an environment at a first capture position and first orientation and second three-dimensional data captured from the object or the environment at a second capture position and second orientation, wherein the first three-dimensional data and the second three-dimensional data respectively comprise spatial features associated with different non-overlapping volumes respectively corresponding to different non-overlapping portions of the object or the environment;
determining a first set of visual features included in a first two-dimensional image associated with the first three-dimensional data and captured from the first capture position and first orientation;
determining a second set of visual features included in a second two-dimensional image associated with the second three-dimensional data and associated with one or more of the spatial features that are included in the second three-dimensional data;
identifying pairs of corresponding features respectively comprising first features included in the first set of visual features and second features included in the second set of visual features;
determining spatial distances between respective features of the pairs of corresponding features based on known positions of the second features relative to the three-dimensional coordinate space; and
determining an alignment score representative of a quality of the alignment based on the spatial distances.

12. The system of claim 11 , wherein the operations further comprise:
employing, by the system, the alignment score to generate an optimized alignment between the first three-dimensional data and the second three-dimensional data.

13. The system of claim 11 , wherein the identifying the pairs of corresponding features comprises identifying the pairs of corresponding features based on a visual correspondence requirement associated with visual correspondence between the respective featur

Assignees

• Matterport Inc

Inventors

• Bell Matthew
• Gausebeck David
• Beebe Michael

Classifications

• H04N13/106 Primary

  Processing image signals (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/10

  Processing, recording or transmission of stereoscopic or multi-view image signals · CPC title

• H04N13/246

  Calibration of cameras · CPC title

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title