Methods and apparatus for merging depth images generated using distinct depth imaging techniques

Patent metadata

Field	Value
Publication number	US-2016005179-A1
Application number	US-201314233943-A
Country	US
Kind code	A1
Filing date	Aug 23, 2013
Priority date	Dec 17, 2012
Publication date	Jan 7, 2016
Grant date	—

Abstract

A depth imager is configured to generate a first depth image using a first depth imaging technique, and to generate a second depth image using a second depth imaging technique different than the first depth imaging technique. At least portions of the first and second depth images are merged to form a third depth image. The depth imager comprises at least one sensor including a single common sensor at least partially shared by the first and second depth imaging techniques, such that the first and second depth images are both generated at least in part using data acquired from the single common sensor. By way of example, the first depth image may comprise a structured light (SL) depth map generated using an SL depth imaging technique, and the second depth image may comprise a time of flight (ToF) depth map generated using a ToF depth imaging technique.

First claim

What is claimed is:

1 . A method comprising:
generating a first depth image using a first depth imaging technique; generating a second depth image using a second depth imaging technique different than the first depth imaging technique; and merging at least portions of the first and second depth images to form a third depth image; wherein the first and second depth images are both generated at least in part using data acquired from a single common sensor of a depth imager.

2 . The method of claim 1 wherein the first depth image comprises a structured light depth map generated using a structured light depth imaging technique, and the second depth image comprises a time of flight depth map generated using a time of flight depth imaging technique.

3 . The method of claim 1 wherein the first and second depth images are generated at least in part using respective first and second different subsets of a plurality of sensor cells of the single common sensor.

4 . The method of claim 1 wherein the first depth image is generated at least in part using a designated subset of a plurality of sensor cells of the single common sensor and the second depth image is generated without using the sensor cells of the designated subset.

5 . The method of claim 2 wherein generating the first and second depth images comprises, for a given cell of the common sensor:
receiving amplitude information from the given cell;
demodulating the amplitude information to generate phase information;
generating a time of flight depth estimate using the phase information;
generating a time of flight reliability estimate using the amplitude information;
receiving intensity information from the given cell;
generating a structured light depth estimate using the intensity information; and
generating a structured light reliability estimate using the intensity information.

6 . The method of claim 5 further comprising generating a local depth estimate for the given cell based on the time of flight and structured light depth estimates and the corresponding time of flight and structured light reliability estimates.

7 . The method of claim 5 wherein generating the structured light depth estimate and the corresponding structured light reliability estimate comprises:
generating estimated structured light intensity information using the intensity information;
generating the structured light depth estimate using the estimated structured light intensity information; and
generating the structured light reliability estimate using the intensity information.

8 . The method of claim 5 further comprising generating a global depth estimate for the given cell and one or more additional cells of the sensor based on the time of flight and structured light depth estimates and the corresponding time of flight and structured light reliability estimates as determined for the given cell and similarly determined for the one or more additional cells.

9 . The method of claim 2 wherein generating the first and second depth images comprises:
generating the structured light depth map as a combination of structured light depth information obtained using a first plurality of cells of the common sensor;
generating the time of flight depth map as a combination of time of flight depth information obtained using a second plurality of cells of the common sensor;
preprocessing at least one of the structured light depth map and the time of flight depth map so as to substantially equalize their respective resolutions; and
merging the substantially equalized structured light depth map and the time of flight depth map to generate a merged depth map.

10 . The method of claim 9 wherein said preprocessing comprises:
identifying a particular pixel in the corresponding depth map;
identifying a neighborhood of pixels for the particular pixel; and
interpolating a depth value for the particular pixel based on depth values of the respective pixels in the neighborhood of pixels.

11 . A computer-readable storage medium having computer program code embodied therein, wherein the computer program code when executed in an image processing system comprising the depth imager causes the image processing system to perform the method as recited in claim 1 .

12 . An apparatus comprising:
a depth imager comprising at least one sensor; wherein the depth imager is configured to generate a first depth image using a first depth imaging technique, and to generate a second depth image using a second depth imaging technique different than the first depth imaging technique; wherein at least portions of each of the first and second depth images are merged to form a third depth image; and wherein said at least one sensor comprises a single common sensor at least partially shared by the first and second depth imaging techniques such that the first and second depth images are both generated at least in part using data acquired from the single common sensor.

13 . The apparatus of claim 12 wherein the first depth image comprises a structured light depth map generated using a structured light depth imaging technique, and the second depth image comprises a time of flight depth map generated using a time of flight depth imaging technique.

14 . The apparatus of claim 12 wherein the depth imager further comprises a first emitter configured to generate output light in accordance with a structured light depth imaging technique and a second emitter configured to generate output light in accordance with a time of flight depth imaging technique.

15 . The apparatus of claim 12 wherein the depth imager comprises at least one emitter wherein said at least one emitter comprises a single common emitter configured to generate output light in accordance with both a structured light depth imaging technique and a time of flight depth imaging technique.

16 . The apparatus of claim 12 wherein the depth imager is configured to generate the first and second depth images at least in part using respective first and second different subsets of a plurality of sensor cells of the single common sensor.

17 . The apparatus of claim 12 wherein the depth imager is configured to generate the first depth image at least in part using a designated subset of a plurality of sensor cells of the single common sensor and to generate the second depth image without using the sensor cells of the designated subset.

18 . The apparatus of claim 12 wherein the single common sensor comprises a plurality of structured light sensor cells and a plurality of time of flight sensor cells.

19 . The apparatus of claim 12 wherein the single common sensor comprises at least one sensor cell that is a joint structured light and time of flight sensor cell.

20 . An image processing system comprising:
at least one processing device; and a depth imager associated with the processing device and comprising at least one sensor; wherein the depth imager is configured to generate a first depth image using a first depth imaging technique, and to generate a second depth image using a second depth imaging technique different than the first depth imaging technique; wherein at least portions of each of the first and second depth images are merged to form a third depth image; and wherein said at least one sensor comprises a single common sensor at least partially shared by the first and second depth imaging techniques such that the first and second depth images are both generated at least in part using data acquired from the

Assignees

• Lsi Corp

Inventors

• Petyushko Alexander A
• Parfenov Denis V
• Mazurenko Ivan L
• Kholodenko Alexander B

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20228

  Disparity calculation for image-based rendering · CPC title

• H04N13/025

  Electricity · mapped topic

• H04N13/0271

  Electricity · mapped topic

• G06T7/0065 Primary

  Physics · mapped topic