System and method of capturing and generating panoramic three-dimensional images

Patent metadata

Field	Value
Publication number	US-11640000-B2
Application number	US-202217664602-A
Country	US
Kind code	B2
Filing date	May 23, 2022
Priority date	Dec 30, 2019
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

An apparatus comprising a housing, a mount configured to be coupled to a motor to horizontally move the apparatus, a wide-angle lens coupled to the housing, the wide-angle lens being positioned above the mount thereby being along an axis of rotation, the axis of rotation being the axis along which the apparatus rotates, an image capture device within the housing, the image capture device configured to receive two-dimensional images through the wide-angle lens of environment, and a LiDAR device within the housing, the LiDAR device configured to generate depth data based on the environment.

First claim

The invention claimed is:

1. An image capture device comprising:
a housing, the housing having a front side and a back side;
a first motor coupled to the housing at a first position between the front side and the back side of the housing, the first motor being configured to horizontally turn the image capture device substantially 270 degrees about a vertical axis;
a wide-angle lens coupled to the housing of the image capture device at a second position between the front side and the back side of the housing along the vertical axis, the second position being a no-parallax point, the wide-angle lens having a field of view away from the front side of the housing;
an image sensor the housing of the image capture device and configured to generate image signals from light received by the wide-angle lens;
a mount coupled to the first motor;
a LiDAR coupled to the housing of the image capture device at a third position, the LiDAR configured to generate laser pulses and generate depth signals;
a second motor coupled to the housing of the image capture device; and
a mirror coupled to the second motor, the second motor configured to rotate the mirror around a horizontal axis, the mirror including an angled surface configured to receive the laser pulses from the LiDAR and direct the laser pulses about the horizontal axis.

2. The image capture device of claim 1 , wherein the image sensor is configured to generate a first plurality of images at different exposures when the image capture device is stationary and pointed in a first direction.

3. The image capture device of claim 2 , wherein the first motor is configured to turn the image capture device about the vertical axis after the first plurality of images are generated.

4. The image capture device of claim 3 , wherein the image sensor does not generate images while the first motor turns the image capture device and wherein the LiDAR generates depth signals based on the laser pulses while the first motor turns the image capture device.

5. The image capture device of claim 3 , wherein the image sensor is configured to generate a second plurality of images at the different exposures when the image capture device is stationary and pointed in a second direction and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after the second plurality of images are generated.

6. The image capture device of claim 5 , wherein the image sensor is configured to generate a third plurality of images at the different exposures when the image capture device is stationary and pointed in a third direction and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after the third plurality of images are generated.

7. The image capture device of claim 6 , wherein the image sensor is configured to generate a fourth plurality of images at the different exposures when the image capture device is stationary and pointed in a fourth direction and the first motor is configured to turn the image capture device 90 degrees about the vertical axis after the fourth plurality of images are generated.

8. The image capture device of claim 7 , further comprising a processor configured to blend frames of the first plurality of images before the image sensor generates the second plurality of images.

9. The image capture device of claim 7 , further comprising a remote digital device in communication with the image capture device and configured to generate a 3D visualization based on the first, second, third, and fourth plurality of images and the depth signals, the remote digital device being configured to generate the 3D visualization using no more images than the first, second, third, and fourth plurality of images.

10. The image capture device of claim 9 , wherein first, second, third, and fourth plurality of images are generated between turns that combined turns turning the image capture device 270 degrees around the vertical axis.

11. The image capture device of claim 4 , wherein a speed or rotation of the mirror around the horizontal axis increases as the first motor turns the image capture device.

12. The image capture device of claim 1 , wherein the angled surface of the mirror is 90 degrees.

13. The image capture device of claim 1 wherein the LiDAR emits the laser pulses in a direction that is opposite the front side of the housing of the image capture device.

14. A method comprising:
receiving light from a wide-angle lens of an image capture device, the wide-angle lens being coupled to a housing of the image capture device, the light being received at a field of view of the wide-angle lens, the field of view extending away from a front side of the housing;
generating a first plurality of images by an image sensor of an image capture device using the light from the wide-angle lens, the image sensor being coupled to the housing of the image capture device, the first plurality of images being at different exposures;
horizontally turning the image capture device by a first motor substantially 270 degrees about a vertical axis, the first motor being coupled to the housing in a first position between the front side and a back side of the housing, the wide-angle lens being at a second position along the vertical axis, the second position being a no-parallax point;
rotating a mirror with an angled surface around horizontal axis by a second motor, the second motor being within coupled to the housing of the image capture device;
generating laser pulses by a LiDAR, the LiDAR being within the housing of the image capture device at a third position, the laser pulse being directed to the rotating mirror while the image capture device horizontally turns; and
generating depth signals by the LiDAR based on the laser pulses.

15. The method of claim 14 , wherein generating the first plurality of images by the image sensor occurs before the image captured device horizontally turns.

16. The method of claim 15 , wherein the image sensor does not generate images while the first motor turns the image capture device and wherein the LiDAR generates the depth signals based on the laser pulses while the first motor turns the image capture device.

17. The method of claim 16 , further comprising:
generating a second plurality of images at the different exposures by the image sensor when the image capture device is stationary and pointed in a second direction; and
turning the image capture device 90 degrees about the vertical axis by the first motor after the second plurality of images are generated.

18. The method of claim 17 , further comprising:
generating a third plurality of images at the different exposures by the image sensor when the image capture device is stationary and pointed in a third direction; and
turning the image capture device 90 degrees about the vertical axis by the first motor after the third plurality of images are generated.

19. The method of claim 18 , further comprising:
generating a fourth plurality of images at the different exposures by the image sensor when the image capture device is stationary and pointed in a fourth direction.

20. The method of claim 19 , further comprising generating a 3D visualization using the first, second, third, and fourth plurality of images and based on the depth signals, the generating the 3D visualization not using any other images.

21. The method of claim 17 , further comprising blending frames of the first plurality of images before the image sensor generat

Assignees

• Matterport Inc

Inventors

• Gausebeck David Alan
• Stromberg Kirk
• Marzano Louis D
• Proctor David
• Sakakibara Naoto
• Trieu Simeon
• Kane Kevin
• Wynn Simon

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G03B17/561

  Support related camera accessories · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G01S7/4817

  relating to scanning · CPC title

• G01S17/931 Primary

  of land vehicles · CPC title