Depth map generation and post-capture focusing

Patent metadata

Field	Value
Publication number	US-9294662-B2
Application number	US-201314065786-A
Country	US
Kind code	B2
Filing date	Oct 29, 2013
Priority date	Oct 16, 2013
Publication date	Mar 22, 2016
Grant date	Mar 22, 2016

Abstract

Aspects of depth map generation and post capture focusing and re-focusing are described. According to one embodiment, a depth map is generated. The depth map may include a mapping among relative depth values in a field of view of an image based on a difference between pixels of a first image and pixels of a second image. An edge map may also be generated by identifying edges in at least one of the first image or the second image. Using the depth map and the edge map, the relative depth values in the depth map may be smoothed using the edge map. In this manner, certain discontinuities in depth values may be smoothed within edge-bounded regions defined by the edge map. The depth map may be used for focusing and re-focusing, for example, or for object extraction, scene understanding, or gesture recognition, among other imaging processes.

First claim

Therefore, at least the following is claimed:

1. An image processing method, comprising:
generating, with a processing circuit, a depth map including a mapping among relative depth values in a field of view based on a difference between pixels of a first image and pixels of a second image;
generating, with the processing circuit, an edge map identifying edges in at least one of the first image or the second image; and
smoothing, with the processing circuit, the relative depth values of the depth map based at least in part on a linear fit of the relative depth values between the edges identified by the edge map.

2. The method of claim 1 , further comprising:
capturing the first image and the second image, a pixel density of the second image being a scalar fraction of the pixel density of the first image; and
before generating the edge map, downscaling the first image to a downscaled first image, a pixel density of the downscaled first image being substantially equivalent to the pixel density of the second image.

3. The method of claim 1 , further comprising:
receiving a point for focus within the field of view; and
focusing the first image according to the point for focus with reference to the relative depth values of the depth map.

4. The method of claim 3 , wherein focusing the first image comprises determining a point spread per pixel for one or more pixels of the first image with reference to the relative depth values of the depth map and the point for focus.

5. The method of claim 3 , wherein focusing the first image comprises:
generating a blurred replica of the first image; and
blending the blurred replica of the first image with the first image according to the relative depth values of the depth map and the point for focus.

6. The method of claim 3 , wherein focusing the first image comprises focus stacking regions of the first image with regions of a plurality of images having different focus points or depths of field with reference to the relative depth values of the depth map.

7. The method of claim 1 , comprising:
capturing the first image with a first sensor; and
capturing the second image with a second sensor while capturing the first image with the first sensor.

8. The method of claim 1 , wherein smoothing the relative depth values of the depth map comprises:
filtering the relative depth values in a column of the depth map between a first pair of edges in the edge map; and
filtering the relative depth values in a row of the depth map between a second pair edges in the edge map.

9. The method of claim 1 , wherein smoothing the relative depth values of the depth map further comprises iteratively filtering, in rows and columns, depth values of the depth map between pairs of edges in the edge map.

10. An image processing device,
comprising: at least one sensor;
a memory coupled to the at least one sensor; and
a processing circuit coupled to the memory and configured to:
generate a depth map including a mapping among relative depth values in a field of view of at least one of a first image or a second image;
generate an edge map by identifying edges in at least one of the first image or the second image; and
smooth the relative depth values of the depth map based at least in part on a linear fit of the relative depth values between the edges identified by the edge map.

11. The image processing device of claim 10 , wherein the processing circuit is further configured to generate the depth map based on a difference between pixels of the first image and pixels of the second image.

12. The image processing device of claim 10 , wherein:
the at least one sensor captures the first image and the second image; and
the processing circuit is further configured to downscale at least one of the first image or the second image before the depth map and the edge map are generated.

13. The image processing device of claim 10 , wherein the processing circuit is further configured to:
receive a point for focus within a field of view of the first image or the second image; and
focus the first image according to the point for focus with reference to the relative depth values of the depth map.

14. The image processing device of claim 13 , wherein the processing circuit is further configured to determine a point spread per pixel for one or more pixels of the first image with reference to the relative depth values of the depth map and the point for focus.

15. The image processing device of claim 13 , wherein the processing circuit is further configured to:
generate a blurred replica of the first image; and
blend the blurred replica of the first image with the first image according to the relative depth values of the depth map and the point for focus.

16. The image processing device of claim 13 , wherein the processing circuit is further configured to focus stack regions of the first image with regions of a plurality of images having different focus points or depths of field with reference to the relative depth values of the depth map.

17. The image processing device of claim 10 , wherein the at least one sensor comprises a first sensor that captures the first image, and a second sensor that captures the second image while the first image is captured with the first sensor.

18. A computer program product comprising instructions stored in a non-transitory computer-readable storage medium, the instructions comprising:
instructions to capture a first image and a second image using a mix of image sensors;
instructions to generate a depth map including a mapping among relative depth values based on a difference between pixels of the first image and pixels of the second image;
instructions to generate an edge map by identifying edges in at least one of the first image or the second image; and
instructions to smooth, the relative depth values of the depth map based at least in part on a linear fit of the relative depth values between the edges identified by the edge map.

19. The computer program product of claim 18 , wherein the instructions further comprise:
instructions to receive a point for focus; and
instructions to focus the first image according to the point for focus with reference to the relative depth values of the depth map.

20. The computer program product of claim 19 , wherein the instructions to focus the first image comprises:
instructions to generate a blurred replica of the first image; and
instructions to blend the blurred replica of the first image with the first image according to the relative depth values of the depth map and the point for focus.

Assignees

• Broadcom Corp

Inventors

• Vondran Jr Gary Lee
• Macfarlane Charles Dunlop

Classifications

• H04N17/002 Primary

  for television cameras · CPC title

• H04N23/13

  with multiple sensors · CPC title

• H04N23/45 Primary

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title