Combining light-field data with active depth data for depth map generation

Patent metadata

Field	Value
Publication number	US-11328446-B2
Application number	US-201715635894-A
Country	US
Kind code	B2
Filing date	Jun 28, 2017
Priority date	Apr 15, 2015
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Depths of one or more objects in a scene may be measured with enhanced accuracy through the use of a light-field camera and a depth sensor. The light-field camera may capture a light-field image of the scene. The depth sensor may capture depth sensor data of the scene. Light-field depth data may be extracted from the light-field image and used, in combination with the sensor depth data, to generate a depth map indicative of distance between the light-field camera and one or more objects in the scene. The depth sensor may be an active depth sensor that transmits electromagnetic energy toward the scene; the electromagnetic energy may be reflected off of the scene and detected by the active depth sensor. The active depth sensor may have a 360° field of view; accordingly, one or more mirrors may be used to direct the electromagnetic energy between the active depth sensor and the scene.

First claim

What is claimed is:

1. A method for measuring depths of at least one object in a scene, the method comprising:
with a light-field camera, capturing a light-field image of the scene;
with a depth sensor, capturing depth sensor data of the scene;
extracting light-field depth data from the captured light-field image;
combining the extracted light-field depth data with the captured depth sensor data to generate a depth map indicative of distance between the light-field camera and each object in the scene by:
calibrating the light-field camera, depth sensor, or both to generate a correspondence between the sensor depth data and the light-field image, wherein calibrating the light-field camera, depth sensor, or both comprises:
with the depth sensor, capturing depth calibration data of a calibration scene containing a planar board positioned at one or more known orientations;
capturing light-field calibration data of the calibration scene contemporaneously with capture of the depth calibration data;
using the depth calibration data to ascertain locations of corners of the planar board in each of one or more known orientations; and
establishing settings for the light-field camera in which the corners in the light-field calibration data are aligned with the corners in the depth calibration data;

creating a 3D data cost function indicating a likelihood of a depth for at least a portion of the scene based on multi-view geometry;
applying cross-based aggregation heuristics to improve the 3D data cost function; and
applying at least one of local and global optimization to generate the depth map; and

outputting the generated depth map.

2. The method of claim 1 , wherein:
the depth sensor comprises an active depth sensor;
the method further comprises, prior to capturing the depth sensor data, transmitting electromagnetic energy toward the scene with the active depth sensor; and
capturing the depth sensor data comprises:
receiving the electromagnetic energy after reflection of the electromagnetic energy off of the scene; and
based on receipt of the electromagnetic energy, generating the depth sensor data.

3. The method of claim 2 , wherein the active depth sensor comprises a LiDAR sensor.

4. The method of claim 2 , wherein the active depth sensor comprises a Time of Flight sensor.

5. The method of claim 2 , wherein:
the active depth sensor has a 360° field of view;
transmitting the electromagnetic energy toward the scene comprises:
with the active depth sensor, emitting electromagnetic energy generally radially outward from the active depth sensor; and
with one or more mirrors, reflecting the emitted electromagnetic energy toward the scene; and
receiving the electromagnetic energy comprises, with the one or more mirrors, reflecting the emitted electromagnetic energy generally radially inward toward the active depth sensor.

6. The method of claim 5 , wherein the one or more mirrors are arranged to define a conical shape.

7. The method of claim 5 , wherein the one or more mirrors are arranged to define a pyramidal shape.

8. The method of claim 5 , wherein the one or more mirrors are arranged to define a multi-faceted shape comprising more than four facets.

9. The method of claim 1 , wherein using the light-field depth data and the depth sensor data to generate the depth map further comprises applying a depth map generation algorithm to combine the sensor depth data with the light-field depth data.

10. The method of claim 1 , wherein using the light-field depth data and the depth sensor data to generate the depth map further comprises: modifying the 3D data cost function to accommodate a plurality of depth sensor samples from the depth sensor data of the scene.

11. A non-transitory computer-readable medium for measuring depths of at least one object in a scene, comprising instructions stored thereon, that when executed by a processor, perform the steps of:
receiving a light-field image of a scene, wherein the light-field image has been captured with a light-field camera;
receiving depth sensor data of the scene, wherein the depth sensor data has been captured with a depth sensor;
extracting light-field depth data from the captured light-field image;
combining the extracted light-field depth data with the captured depth sensor data to generate a depth map indicative of distance between the light-field camera and each object in the scene by:
calibrating the light-field camera, depth sensor, or both to generate a correspondence between the sensor depth data and the light-field image, wherein calibrating the light-field camera, depth sensor, or both comprises:
with the depth sensor, capturing depth calibration data of a calibration scene containing a planar board positioned at one or more known orientations;
capturing light-field calibration data of the calibration scene contemporaneously with capture of the depth calibration data;
using the depth calibration data to ascertain locations of corners of the planar board in each of one or more known orientations; and
establishing settings for the light-field camera in which the corners in the light-field calibration data are aligned with the corners in the depth calibration data;

creating a 3D data cost function based on multi-view geometry;
applying cross-based aggregation heuristics to improve the 3D data cost function; and
applying at least one of local and global optimization to generate the depth map; and

outputting the generated depth map.

12. The non-transitory computer-readable medium of claim 11 , wherein using the light-field depth data and the depth sensor data to generate the depth map further comprises: modifying the 3D data cost function to accommodate a plurality of depth sensor samples from the depth sensor data of the scene.

13. A system for measuring depths of at least one object in a scene, the system comprising:
a light-field camera configured to capture a light-field image of the scene;
a depth sensor configured to capture depth sensor data of the scene; and
a processor, communicatively coupled to the light-field camera and the depth sensor, configured to:
extract light-field depth data from the captured light-field image; and
combine the extracted light-field depth data with the captured depth sensor data to generate a depth map indicative of distance between the light-field camera and each object in the scene by:
calibrating the light-field camera, depth sensor, or both to generate a correspondence between the sensor depth data and the light-field image, wherein calibrating the light-field camera, depth sensor, or both comprises:
with the depth sensor, capturing depth calibration data of a calibration scene containing a planar board positioned at one or more known orientations;
capturing light-field calibration data of the calibration scene contemporaneously with capture of the depth calibration data;
using the depth calibration data to ascertain locations of corners of the planar board in each of one or more known orientations; and
establishing settings for the light-field camera in which the corners in the light-field calibration data are aligned with the corners in the depth calibration data;

creating a 3D data cost function indicating a likelihood of a depth for at least a portion of the scene based on multi-view geometry;
applying cross-based aggregation heuristics to improve the 3D data cost function; and
applying at least one of local and global optimization to generate the depth map; and

an output device, communicatively coupled to the processor, configured to output the generated depth map.

14. The syst

Assignees

• Google Llc

Inventors

• Tan Jie
• Pan Gang
• Karafin Jon
• Nonn Thomas
• Hernandez Zaragoza Julio C

Classifications

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• H04N13/232 Primary

  using fly-eye lenses, e.g. arrangements of circular lenses · CPC title

• G01S17/42

  Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) · CPC title

• G01S17/08

  for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title