Capturing and aligning three-dimensional scenes

Patent metadata

Field	Value
Publication number	US-10529142-B2
Application number	US-201314070428-A
Country	US
Kind code	B2
Filing date	Nov 1, 2013
Priority date	Feb 24, 2012
Publication date	Jan 7, 2020
Grant date	Jan 7, 2020

Abstract

Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor.

First claim

What is claimed is:

1. A device, comprising:
a processor, and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
receiving three-dimensional data of an environment captured via a three-dimensional capture device of the device;
determining an alignment between the three-dimensional data and a three-dimensional model of the environment;
based on the alignment, determining a current position of the device relative to a first location on or within the three-dimensional model that is associated with an auxiliary data object, wherein the auxiliary data object comprises visual data capable of being presented on a display of the device; and
based on the current position of the device relative to the first location and a current viewpoint of the environment as viewed on or through the display, determining a perspective for displaying the auxiliary data object via the display that spatially aligns the auxiliary data object with a second location in the environment that corresponds to the first location on or within the three-dimensional model.

2. The device of claim 1 , wherein the operations further comprise:
displaying the auxiliary data object via the display based on the perspective.

3. The device of claim 1 , wherein the determining the alignment comprises determining the alignment by matching first spatial features of the three-dimensional data with second spatial features of the three-dimensional model.

4. The device of claim 1 , wherein the current viewpoint of the environment comprises a viewpoint of a user of the device relative to the environment and the display, and wherein the operations further comprise:
determining the current viewpoint based on image data captured of the user's face.

5. The device of claim 2 , wherein the displaying comprises overlaying the auxiliary data object over a live image of the environment using depth-based occlusion.

6. The device of claim 2 , wherein the operations further comprise:
generating a three-dimensional rendering comprising the auxiliary data object from the perspective by combining the auxiliary data object with the three-dimensional model at the first location on or within the three dimensional-model, and wherein the displaying comprises displaying the three-dimensional rendering.

7. The device of claim 2 , wherein the operations further comprise:
generating another three-dimensional model of the environment based on the three-dimensional data; and
generating a three-dimensional rendering comprising the auxiliary data object from the perspective by combining the other three-dimensional model with the auxiliary data object, and wherein the displaying comprises displaying the three-dimensional rendering.

8. The device of claim 1 , wherein the three-dimensional model of the environment is generated at a substantially same time as the the determining the alignment between the three-dimensional data of the environment and the three-dimensional model.

9. The device of claim 1 , wherein the three-dimensional model of the environment is generated and the auxiliary data is captured at a substantially same time as the the determining the alignment between the three-dimensional data of the environment and the three-dimensional model.

10. A method, comprising:
receiving, by a system comprising a processor, three-dimensional data captured of an environment by a three-dimensional capture device;
determining, by the system, an alignment between the three-dimensional data and a three-dimensional reconstruction of the environment;
based on the alignment, determining, by the system, a current position of the three-dimensional capture device relative to a first location on or within the three-dimensional reconstruction that is associated with auxiliary data, wherein the auxiliary data comprises visual data capable of being presented on a display; and
based on the current position of the three-dimensional capture device relative to the first location and a current viewpoint of the environment as viewed on or through the display, determining, by the system, a position for displaying the auxiliary data via the display that spatially aligns the auxiliary data with a second location in the environment that corresponds to the first location on or within the three-dimensional reconstruction.

11. The method of claim 10 , further comprising:
generating, by the system, a three-dimensional rendering comprising the auxiliary data at the position.

12. The method of claim 10 , further comprising:
displaying, by the system, the auxiliary data via the display.

13. The method of claim 12 , wherein the display comprises a transparent or partially transparent display and wherein the displaying comprises overlaying the auxiliary data object over a live view of the environment viewed through the display.

14. The method of claim 10 , wherein the current viewpoint of the environment comprises a viewpoint of a user relative to the environment and the display, the method further comprising:
receiving, by the system, image data of the user's face; and
determining, by the system, the current viewpoint based on the image data.

15. The method of claim 10 , wherein the determining the alignment comprises determining the alignment by matching first image or first spatial features of the live three-dimensional data with second image or second spatial features of the three-dimensional reconstruction.

16. The method of claim 11 , wherein the generating the three-dimensional rendering comprises combining the auxiliary data with the three-dimensional data.

17. The method of claim 11 , wherein the generating the three-dimensional rendering comprises combining the auxiliary data with the three-dimensional reconstruction.

18. The method of claim 11 , further comprising:
generating, by the system, another three-dimensional representation of the environment based on the three-dimensional data, wherein the generating the three-dimensional rendering comprises combining the other three-dimensional representation with the auxiliary data to.

19. The method of claim 10 , further comprising:
generating, by the system, the three-dimensional reconstruction at a substantially same time as the determining the alignment.

20. The method of claim 19 , further comprising:
capturing, by the system, the auxiliary data at a substantially same time as the generating and the determining the alignment.

21. The method of claim 10 , further comprising:
capturing, by the system, the auxiliary data via a sensor.

22. The method of claim 21 , wherein the sensor comprises a camera configured to capture surface information associated with the three-dimensional reconstruction.

23. A non-transitory computer-readable storage medium comprising computer-readable instructions that, in response to execution, cause a computing system to perform operations, comprising:
capturing first three-dimensional data of an environment using a three-dimensional capture device;
aligning the first three-dimensional data with a three-dimensional model of the environment generated based on second three-dimensional data of the environment, wherein the three-dimensional model is associated with auxiliary data that is spatially aligned with a first location on or within the three-dimensional model;
determining a current position of the three-dimensional capture device relative to the first position on or wit

Assignees

• Matterport Inc

Inventors

• Bell Matthew
• Gausebeck David
• Beebe Michael

Classifications

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• H04N13/10

  Processing, recording or transmission of stereoscopic or multi-view image signals · CPC title

• H04N2213/001

  Constructional or mechanical details · CPC title

• H04N13/106 Primary

  Processing image signals (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/246

  Calibration of cameras · CPC title