Capturing and aligning three-dimensional scenes

Patent metadata

Field	Value
Publication number	US-12014468-B2
Application number	US-202117165653-A
Country	US
Kind code	B2
Filing date	Feb 2, 2021
Priority date	Feb 24, 2012
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor.

First claim

What is claimed is:

1. A system for building a three-dimensional composite scene, the system comprising:
a three-dimensional capture device for capturing a plurality of three-dimensional images of an object or an environment at different capture positions relative to a three-dimensional coordinate space over a capture process, each of the plurality of three-dimensional images including metadata;
a processor for executing instructions stored in memory, the processor being remote to the three-dimensional capture device, wherein execution of the instructions by the processor includes:
receiving the plurality of three-dimensional images of the object or the environment; and
aligning the received plurality of three-dimensional images to generate a three-dimensional representation of the object or the environment, the aligning being based on the metadata associated with each of the received plurality of three-dimensional images; and

an interface device for displaying at least a part of the three-dimensional representation of the object or the environment, the interface device including a live display area configured to display a live video feed of the three-dimensional representation as it is captured by the three-dimensional capture device, the interface device further including a secondary display area that provides an indication of an area of the object or environment that the three-dimensional capture device should be aimed at to capture a three-dimensional data object of the three-dimensional data objects that facilitates alignment of other three-dimensional data objects of the three-dimensional data objects.

2. The system of claim 1 , wherein the interface device is remote to the three-dimensional capture device.

3. The system of claim 1 , wherein the aligning the received plurality of three-dimensional images is done iteratively.

4. The system of claim 1 , wherein the metadata includes distance information for features included in the plurality of three-dimensional images.

5. The system of claim 4 , wherein the aligning the received plurality of three-dimensional images is based on the distance information for the features respectively included therein and aligning the features to one another in the three-dimensional coordinate space based on the distance information to generate the three-dimensional representations of the object or the environment concurrently with the capture process.

6. The system of claim 1 , wherein the three-dimensional capture device includes a transmitter to wirelessly transmit the plurality of three-dimensional images to the processor.

7. The system of claim 1 , wherein the system further includes a rotatable stage on which the three-dimensional capture device is mounted thereon.

8. A system for building a three-dimensional composite scene, the system comprising:
a three-dimensional capture device for capturing a plurality of three-dimensional images of an object or an environment at different capture positions relative to a three-dimensional coordinate space over a capture process, each of the plurality of three-dimensional images including metadata; and
a processor for executing instructions stored in memory, the processor being remote to the three-dimensional capture device, wherein execution of the instructions by the processor includes:
receiving the plurality of three-dimensional images of the object or the environment;
aligning the received plurality of three-dimensional images to generate a three-dimensional representation of the object or the environment, the aligning being based on the metadata associated with each of the received plurality of three-dimensional images; and
an interface device for displaying at least a part of the three-dimensional representation of the object or the environment, the interface device being configured to generate a prompt in response to a failure to align a subset of three-dimensional data, objects of the three-dimensional data objects, wherein the prompt requests user input identifying an area of or around the three-dimensional representation to attempt realignment.

9. The system of claim 1 , wherein the interface device is configured to identify holes in the three-dimensional representation.

10. A system comprising:
a processor; and
a memory including executable instructions to configure the processor to receive a plurality of three-dimensional images of an object or an environment from a three-dimensional capture device captured by the three-dimensional capture device at different positions relative a three-dimensional coordinate space during a capture process, each of the plurality of three-dimensional images including metadata, and send the plurality of three-dimensional images to a remote processing system as they are respectively received, wherein the remote processing system is configured to align the plurality of three-dimensional images to generate a three-dimensional representation of the object or the environment, the align the plurality of three-dimensional images being based on the metadata associated with each of the plurality of three-dimensional images, sending the three-dimensional representation of the object or the environment to a remote display for rendering, the remote display including a live display area to display a live video feed of the three-dimensional representation of the object or the environment as it is received from the three-dimensional capture device, the remote display further including a secondary display area that provides an indication of an area of the object or environment that the three-dimensional capture device should be aimed at to capture a three-dimensional data object of the three-dimensional data objects that facilitates alignment of other three-dimensional data objects of the three-dimensional data objects.

11. The system of claim 10 , wherein the align the plurality of three-dimensional images is done iteratively.

12. The system of claim 10 , wherein the metadata includes distance information for features included in the plurality of three-dimensional images.

13. The system of claim 12 , wherein the align the received plurality of three-dimensional images is based on the distance information for the features respectively included therein and align the features to one another in the three-dimensional coordinate space based on the distance information to generate the three-dimensional representations of the object or the environment concurrently with the capture process.

14. A method comprising:
capturing, by a three-dimensional capture device, a plurality of three-dimensional images of an object or an environment at different capture positions relative to a three-dimensional coordinate space over a capture process, each of the plurality of three-dimensional images including metadata;
receiving, by a processor remote to the three-dimensional capture device, the plurality of three-dimensional images of the object or the environment;
aligning the received plurality of three-dimensional images to generate a three-dimensional representation of the object or the environment, the aligning being based on the metadata associated with each of the received plurality of three-dimensional images;
displaying, by an interface device, at least a part of the three-dimensional representation of the object or the environment, the interface device including a live display area configured to display a live video feed of the three-dimensional representation as it is captured by the three-dimensional capture device; and
providing, by a secondary display area of the interface device, an indication of an area of the object or environment

Assignees

• Matterport Inc

Inventors

• Bell Matthew
• Gausebeck David
• Beebe Michael

Classifications

• H04N2213/001

  Constructional or mechanical details · CPC title

• H04N13/246

  Calibration of cameras · CPC title

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• H04N13/156

  Mixing image signals · CPC title

• H04N13/106 Primary

  Processing image signals (for multi-view video sequence encoding H04N19/597) · CPC title