Depth mapping based on pattern matching and stereoscopic information

Patent metadata

Field	Value
Publication number	US-10152801-B2
Application number	US-201615271398-A
Country	US
Kind code	B2
Filing date	Sep 21, 2016
Priority date	Jul 30, 2009
Publication date	Dec 11, 2018
Grant date	Dec 11, 2018

Abstract

A method for depth mapping includes projecting a pattern of optical radiation onto an object. A first image of the pattern on the object is captured using a first image sensor, and this image is processed to generate pattern-based depth data with respect to the object. A second image of the object is captured using a second image sensor, and the second image is processed together with another image to generate stereoscopic depth data with respect to the object. The pattern-based depth data is combined with the stereoscopic depth data to create a depth map of the object.

First claim

The invention claimed is:

1. A method for depth mapping, comprising:
projecting a pattern of infrared optical radiation onto an object;
capturing an infrared image of the pattern on the object using a first image sensor, and processing the infrared image alone to generate pattern-based depth data with respect to the object;
capturing a color image of the object using a second image sensor, at a known spacing from the first image sensor, wherein the projected pattern does not appear in the color image, and stereoscopically measuring a local offset between locations in the color image and the infrared image resulting from parallax between the first and second image sensors so as to generate stereoscopic depth data with respect to the object based on the measured offset and the known spacing; and
combining the pattern-based depth data from a first area of a scene containing the object with the stereoscopic depth data from a different, second area of the scene in which the projected pattern has a contrast too low to enable reliable detection to create a depth map of the object.

2. The method according to claim 1 , wherein the color image comprises pixels, and the depth map comprises depth values, and wherein the method comprises outputting the color image to a display together with the depth coordinates that are associated with the pixels.

3. The method according to claim 1 , wherein projecting the pattern comprises projecting multiple spots onto the object, and wherein processing the infrared image comprises finding respective transverse shifts between the spots on the object and the spots in a reference image of the pattern, and computing the depth data based on the transverse shifts.

4. The method according to claim 1 , wherein combining the pattern-based depth data with the stereoscopic depth data comprises computing respective measures of confidence associated with the pattern-based depth data and stereoscopic depth data, and selecting depth coordinates from among the pattern-based and stereoscopic depth data responsively to the respective measures of confidence.

5. The method according to claim 1 , wherein combining the pattern-based depth data with the stereoscopic depth data comprises defining multiple candidate depth coordinates for each of a plurality of pixels in the depth map, and selecting one of the candidate depth coordinates at each pixel for inclusion in the depth map.

6. The method according to claim 5 , wherein the multiple candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

7. The method according to claim 5 , wherein selecting the one of the candidate depth coordinates comprises applying weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

8. The method according to claim 1 , wherein combining the pattern-based depth data with the stereoscopic depth data comprises applying a calibration procedure to the infrared and color images so as to correct for a misalignment between the infrared and color images.

9. The method according to claim 8 , wherein applying the calibration procedure comprises correcting for a change in alignment between the pattern of optical radiation and the first image sensor.

10. Apparatus for depth mapping, comprising:
an illumination subassembly, which is configured to project a pattern of infrared optical radiation onto an object;
a first image sensor, which is configured to capture an infrared image of the pattern on the object;
a second image sensor, which is located at a known spacing from the first image sensor and is configured to capture a color image of the object, wherein the projected pattern does not appear in the color image; and
a processor, which is configured to process the infrared image alone to generate pattern-based depth data with respect to the object, to stereoscopically measure a local offset between locations in the infrared image and the color image resulting from parallax between the first and second image sensors so as to generate stereoscopic depth data with respect to the object based on the measured offset and the known spacing, and to combine the pattern-based depth data from a first area of a scene containing the object with the stereoscopic depth data from a different, second area of the scene in which the projected pattern has a contrast too low to enable reliable detection to create a depth map of the object.

11. The apparatus according to claim 10 , wherein the color image comprises pixels, and the depth map comprises depth values, and wherein the processor is configured to output the color image to a display together with the depth coordinates that are associated with the pixels.

12. The apparatus according to claim 10 , wherein the projected pattern comprises multiple spots that are projected onto the object, and wherein the processor is configured to find respective transverse shifts between the spots on the object and the spots in a reference image of the pattern, and to compute the depth data based on the transverse shifts.

13. The apparatus according to claim 10 , wherein the processor is configured to associate respective measures of confidence with the pattern-based depth data and stereoscopic depth data, and to select depth coordinates from among the pattern-based and stereoscopic depth data responsively to the respective measures of confidence.

14. The apparatus according to claim 10 , wherein the processor is configured to define multiple candidate depth coordinates for each of a plurality of pixels in the depth map, and to select one of the candidate depth coordinates at each pixel for inclusion in the depth map.

15. The apparatus according to claim 14 , wherein the multiple candidate depth coordinates comprise, for at least some of the pixels, a null coordinate indicating that no valid depth coordinate was found.

16. The apparatus according to claim 14 , wherein the processor is configured to apply weighted tensor voting among the pixels in order to select the one of the candidate depth coordinates based on the candidate depth coordinates at neighboring pixels.

17. The apparatus according to claim 10 , wherein the processor is configured to apply a calibration procedure to the infrared and color images so as to correct for a misalignment between the infrared and color images.

18. The apparatus according to claim 17 , wherein the calibration procedure comprises correcting for a change in alignment between the pattern of optical radiation and the first image sensor.

19. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a processor, cause the processor to receive an infrared image from a first image sensor of a pattern that has been projected onto an object and to receive a color image of the object from a second image sensor at a known spacing from the first image sensor, wherein the projected pattern does not appear in the color image, and to process the infrared image alone to generate pattern-based depth data with respect to the object, to stereoscopically measure a local offset between locations in the infrared image and the color image resulting from parallax between the first and second image sensors so as to generate stereoscopic depth data with respect to the object based on the measured offset and the known spacing, and to combine the pattern-based dep

Assignees

• Apple Inc

Inventors

• Shpunt Alexander
• Medioni Gerard
• Cohen Daniel
• Sali Erez
• Deitch Ronen

Classifications

• H04N13/25

  using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/254

  in combination with electromagnetic radiation sources for illuminating objects · CPC title