Systems and methods for capturing and generating panoramic three-dimensional models and images

Patent metadata

Field	Value
Publication number	US-11943539-B2
Application number	US-202217744539-A
Country	US
Kind code	B2
Filing date	May 13, 2022
Priority date	Dec 30, 2019
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

An environmental capture system (ECS) captures image data and depth information in a 360-degree scene. The captured image data and depth information can be used to generate a 360-degree scene. The ECS comprises a frame, a drive train mounted to the frame, and an image capture device coupled to the drive train to capture, while pointed in a first direction, a plurality of images at different exposures in a first field of view (FOV) of the 360-degree scene. The ECS further comprises a depth information capture device coupled to the drive train. The depth information capture device and the image capture device are rotated by the drive train about a first, substantially vertical, axis from the first direction to a second direction. The depth information capture device, while being rotated from the first direction to the second direction, captures depth information for a first portion of the 360-degree scene. The image capture device captures, while pointed in the second direction, a plurality of images at different exposures in a second FOV that overlaps the first FOV of the 360-degree scene. The depth information capture device and the image capture device are rotated by the drive train about the first axis from the second direction to a third direction. The depth information capture device, while being rotated from the second direction to the third direction, captures depth information for a second portion of the 360-degree scene. The image capture device, while pointed in the third direction, captures a plurality of images at different exposures in a third FOV that overlaps the second FOV of the 360-degree scene.

First claim

The invention claimed is:

1. A method for an apparatus to gather images and depth information in a 360-degree scene, wherein the apparatus comprises a frame, a drive train mounted to the frame, a depth information capture device coupled to the drive train, and an image capture device coupled to the drive train, the method comprising:
capturing, by the image capture device while pointed in a first direction, a plurality of images at different exposures in a first field of view (FOV) of the 360-degree scene, the image capture device including a lens situated at a no-parallax point substantially at a center of a first axis;
rotating, by the drive train, the depth information capture device and the image capture device about the first, substantially vertical, axis from the first direction to a second direction;
capturing, by the depth information capture device while being rotated from the first direction to the second direction, depth information for a first portion of the 360-degree scene;
capturing, by the image capture device while pointed in the second direction, a plurality of images at different exposures in a second FOV that overlaps the first FOV of the 360-degree scene;
rotating, by the drive train, the depth information capture device and the image capture device about the first axis from the second direction to a third direction, and capturing, by the depth information capture device while being rotated from the second direction to the third direction, depth information for a second portion of the 360-degree scene;
capturing, by the image capture device while pointed in the third direction, a plurality of images at different exposures in a third FOV that overlaps the second FOV of the 360-degree scene; and
associate depth information for the first, second, and third of the 360-degree scene with numerical coordinates that identify a location of the depth information generated therefrom, the association being based on a distance between the no-parallax point and the depth information capture device.

2. The method of claim 1 , further comprising:
rotating, by the drive train, the depth information capture device and the image capture device about the first axis from the third direction to a fourth direction; and
capturing, by the image capture device while pointed in the fourth direction, a plurality of images at different exposures in a fourth FOV that overlaps the first and the third FOVs of the 360-degree scene.

3. The method of claim 2 , further comprising, while rotating, by the drive train, the depth information capture device and the image capture device about the first axis from the third direction to the fourth direction, capturing, by the depth information capture device, depth information for a third portion of the 360-degree scene.

4. The method of claim 3 , wherein capturing, by the depth information capture device, depth information for the first, second and third portions of the 360-degree scene comprises capturing, by the depth information capture device, depth information for a first, second and third plurality of segments of the 360-degree scene.

5. The method of claim 1 , further comprising:
stitching together, by an image stitching module, one or more of the plurality of images in the first, second and third FOVs into a panoramic image of the 360-degree scene; and
combining, by a 3D image generation module, the depth information for the first, second and third portions of the 360-degree scene with the panoramic image of the 360-degree scene to create the 3D panoramic image of the 360-degree scene.

6. The method of claim 5 , further comprising:
respectively blending, by a blending module, two or more of the plurality of images of the first, second and third FOVs into a first, second and third blended image of the first, second and third FOVs,
wherein stitching together, by the image stitching module, one or more of the plurality of images in the first, second and third FOVs into the panoramic image of the 360-degree scene comprises stitching together the first, second and third blended images into the panoramic image of the 360-degree scene.

7. The method of claim 6 , wherein respectively blending, by the blending module, two or more of the plurality of images of the first, second and third FOVs into the first, second and third blended image of the first, second and third FOVs comprises respectively blending two or more of the plurality of images of the first, second and third FOVs into a first, second and third high dynamic range (HDR) image of the first, second and third FOVs,
wherein stitching together, by the image stitching module, one or more of the plurality of images in the first, second and third FOVs into the panoramic image of the 360-degree scene comprises stitching together the first, second and third HDR images in the first, second and third FOVs into the panoramic image of the 360-degree scene.

8. The method of claim 5 , wherein the depth information capture device comprises a Light Detection and Ranging (lidar) device and a mirror, the method further comprising rotating, by the drive train, the mirror about a second axis, and emitting, by the lidar device, a plurality of laser pulses to the mirror, the mirror, in turn, emitting the plurality of laser pulses in one or more revolutions about a substantially horizontal axis and receiving a corresponding plurality of reflected laser pulses, and the lidar device, in turn, receiving the plurality of reflected laser pulses and generating the depth information therefrom, wherein the first, second and third blended images of the first, second and third FOVs each comprise a plurality of pixels, wherein each of the plurality of pixels is associated with numerical coordinates that identify a location of the pixel in the scene, and wherein each of the reflected laser pulses is likewise associated with corresponding numerical coordinates that identify a location of the depth information generated therefrom, and wherein combining, by the 3D image generation module, the depth information for the 360-degree scene with the panoramic image of the 360-degree scene to create the 3D panoramic image of the 360-degree scene comprises combining the depth information at a location in the 360-degree scene with the panoramic image at the same location in the 360-degree scene to create the 3D panoramic image of the 360-degree scene.

9. The method of claim 1 , wherein the image capture device comprises a lens situated at a no-parallax point substantially at a center of the first axis such that as the drive train rotates the image capture device about the first axis to the first, second and third directions to capture while pointed in those directions the plurality of images at different exposures in the respective first, second and third overlapping FOVs, a parallax effect between the plurality of images in the first, second and third overlapping FOVs is minimized or eliminated.

10. The method of claim 1 , wherein the depth information capture device comprises a Light Detection and Ranging (lidar) device and a mirror, the method further comprising rotating, by the drive train, the mirror about a second axis, and emitting, by the lidar device, a plurality of laser pulses to the mirror, the mirror, in turn, emitting the plurality of laser pulses in one or more revolutions about a substantially horizontal axis and receiving a corresponding plurality of reflected laser pulses, and the lidar device, in turn, receiving the plurality of reflected laser pulses and generating the depth information therefrom, so that as the image capture device and the lidar device are rotated by the drive train about the first axis and the mirror is rotated by the drive train about

Assignees

• Matterport Inc

Inventors

• Gausebeck David Alan
• Stromberg Kirk
• Marzano Louis D
• Proctor David
• Sakakibara Naoto
• Trieu Simeon
• Kane Kevin
• Wynn Simon

Classifications

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N5/2226 Primary

  Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10) · CPC title

• H04N5/265

  Mixing · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title