Building a three-dimensional composite scene

Patent metadata

Field	Value
Publication number	US-9760994-B1
Application number	US-201615292924-A
Country	US
Kind code	B1
Filing date	Oct 13, 2016
Priority date	Jun 29, 2011
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

The capture and alignment of multiple 3D scenes is disclosed. Three dimensional capture device data from different locations is received thereby allowing for different perspectives of 3D scenes. An algorithm uses the data to determine potential alignments between different 3D scenes via coordinate transformations. Potential alignments are evaluated for quality and subsequently aligned subject to the existence of sufficiently high relative or absolute quality. A global alignment of all or most of the input 3D scenes into a single coordinate frame may be achieved. The presentation of areas around a particular hole or holes takes place thereby allowing the user to capture the requisite 3D scene containing areas within the hole or holes as well as part of the surrounding area using, for example, the 3D capture device. The new 3D captured scene is aligned with existing 3D scenes and/or 3D composite scenes.

First claim

What is claimed is:

1. A method comprising:
obtaining, by a system comprising a processor, two or more sets of three dimensional data of a plurality of sets of three dimensional data, wherein the plurality of sets of three dimensional data comprise points in a three dimensional coordinate space, wherein the points are associated with three dimensional features, and wherein the two or more sets are associated with different capture positions relative to the three dimensional coordinate space;
generating, by the system, a plurality of alignment schemes between two or more of the sets of three dimensional data based on the three dimensional data using a spatial transformation;
evaluating, by the system, the plurality of alignment schemes based on quality of alignment of the two or more sets of three dimensional data as aligned with respect to each of the plurality of alignment schemes; and
selecting, by the system, an alignment scheme of the plurality of alignment schemes associated with a higher quality of alignment with respect to other alignment schemes of the plurality of alignment schemes.

2. The method of claim 1 , further comprising:
generating, by the system, a merged set of three dimensional data based on the alignment scheme using the two or more sets of three dimensional data, wherein the merged set of three dimensional data comprises a merged set of three dimensional features based on combined three dimensional features of the two or more sets of three dimensional data.

3. The method of claim 2 , further comprising:
generating, by the system, new alignment schemes between the merged set of three dimensional data and one or more other sets of three dimensional data of the plurality of sets of three dimensional data based at least on the geometric information for the merged set of three dimensional features.

4. The method of claim 2 , further comprising:
evaluating, by the system, the new alignment schemes based on quality of alignment of the merged set of three dimensional data and the one or more other sets of three dimensional data as aligned with respect to each of the new alignment schemes;
selecting, by the system, a new alignment scheme of the new alignment schemes associated with a higher quality of alignment with respect to other new alignment schemes of the new alignment schemes; and
generating, by the system, a new merged set of three dimensional data based on the new alignment scheme using the merged set of three dimensional data and the one or more other sets of three dimensional data.

5. The method of claim 1 , wherein the three dimensional data further comprises appearance information for the three dimensional features, and wherein the evaluating the plurality of alignment schemes comprises comparing geometric information and appearance information for three dimensional features included in the two or more sets of three dimensional data as aligned with respect to each of the plurality of alignment schemes.

6. The method of claim 5 , wherein the evaluating the plurality of alignment schemes further comprises:
determining quality of alignment scores for the plurality of alignment schemes, respectively, based on the comparing, and wherein the selecting the alignment scheme comprises selecting the alignment scheme based in part on a quality of alignment score associated therewith.

7. The method of claim 6 , wherein the determining the quality of alignment scores comprises determining the quality of alignment scores based on consideration of existence of known empty space in the two or more of the plurality of three-dimensional scenes.

8. The method of claim 6 , wherein the determining the quality of alignment scores comprises determining the quality of alignment scores based on types of the features included in the two or more of the plurality of three dimensional scenes.

9. The method of claim 6 , wherein the selecting the alignment scheme comprises selecting the alignment scheme based on association of the alignment scheme with a quality of alignment score above a threshold value.

10. The method of claim 6 , wherein the determining the quality of alignment scores comprises determining the quality of alignment scores based on similarities in geometric location and physical appearance of the features respectively included in the two or more sets of three dimensional data as aligned with respect to each of the plurality of alignment schemes.

11. The method of claim 1 , further comprising, applying a local optimization process to one or more of the plurality of alignment schemes that improves the quality of the alignment of the two or more sets of three dimensional data.

12. The method of claim 1 , further comprising, applying a non-rigid alignment process following the generating the plurality of alignment schemes.

13. The method of claim 1 , further comprising:
receiving, by the system, respective orientation information of a device that captured the plurality of sets of three dimensional data; and
generating, by the system, the plurality of alignment schemes based on the orientation information.

14. The method of claim 1 , further comprising:
receiving, by the system, respective position information of a device that captured the plurality of sets of three dimensional data; and
generating, by the system, the plurality of alignment schemes based on the position information.

15. A system, comprising:
a memory to store executable instructions; and
a processor, coupled to the memory, that facilitates execution of the executable instructions to perform operations, comprising:
receiving two or more sets of three dimensional data, of a plurality of sets of three dimensional data, wherein the plurality of sets of three dimensional data comprise points in a three dimensional coordinate space, wherein the points are associated with three dimensional features, and wherein the two or more sets are associated with different capture positions relative to the three dimensional coordinate space;
generating a plurality of alignment schemes between two or more of the sets of three dimensional data based on the three dimensional data using a spatial transformation;
analyzing the plurality of alignment schemes based on quality of alignment of the two or more sets of three dimensional data as aligned with respect to each of the plurality of alignment schemes; and
selecting an alignment scheme of the plurality of alignment schemes associated with a higher quality of alignment with respect to other alignment schemes of the plurality of alignment schemes.

16. The method of claim 15 , the operations further comprising:
generating a merged set of three dimensional data based on the alignment scheme using the two or more sets of three dimensional data, wherein the merged set of three dimensional data comprises a merged set of three dimensional features based on combined three dimensional features of the two or more sets of three dimensional data.

17. The method of claim 16 , the operations further comprising:
generating new alignment schemes between the merged set of three dimensional data and one or more other sets of three dimensional data of the plurality of sets of three dimensional data based at least on the geometric information for the merged set of three dimensional features.

18. A non-transitory computer-readable medium having instructions stored thereon that, in response to execution, cause a system including a processor to perform operations comprising:
obtaining two or more sets of three dimensional data, of a plurality of sets

Assignees

• Matterport Inc

Inventors

• Bell Matthew
• Gausebeck David

Classifications

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title

• G01S17/89 Primary

  for mapping or imaging · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T7/32

  using correlation-based methods · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title