Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies

Patent metadata

Field	Value
Publication number	US-9519972-B2
Application number	US-201414207254-A
Country	US
Kind code	B2
Filing date	Mar 12, 2014
Priority date	Mar 13, 2013
Publication date	Dec 13, 2016
Grant date	Dec 13, 2016

Abstract

Systems and methods are described for generating restricted depth of field depth maps. In one embodiment, an image processing pipeline application configures a processor to: determine a desired focal plane distance and a range of distances corresponding to a restricted depth of field for an image rendered from a reference viewpoint; generate a restricted depth of field depth map from the reference viewpoint using the set of images captured from different viewpoints, where depth estimation precision is higher for pixels with depth estimates within the range of distances corresponding to the restricted depth of field and lower for pixels with depth estimates outside of the range of distances corresponding to the restricted depth of field; and render a restricted depth of field image from the reference viewpoint using the set of images captured from different viewpoints and the restricted depth of field depth map.

First claim

What is claimed:

1. An image processing system, comprising:
a processor;
memory containing a set of images captured from different viewpoints and an image processing pipeline application;
wherein the image processing pipeline application configures the processor to:
determine a range of distances corresponding to a restricted depth of field; and
generate a restricted depth of field depth map from the reference viewpoint using the set of images captured from different viewpoints, where depth estimation precision is higher for pixels with depth estimates within the range of distances corresponding to the restricted depth of field and lower for pixels with depth estimates outside of the range of distances corresponding to the restricted depth of field;
wherein the image processing pipeline application further configures the processor to generate a restricted depth of field depth map by:
performing a disparity search with respect to a given pixel location using the set of images captured from different viewpoints;
wherein the disparity search is performed using a greater density of depth samples within the range of distances corresponding to the restricted depth of field and a lower density of depth samples for distances outside the range of distances corresponding to the restricted depth of field.

2. The image processing system of claim 1 , wherein the image processing pipeline application further configures the processor to automatically determine the range of distances corresponding to a restricted depth of field.

3. The image processing system of claim 2 , wherein the image processing pipeline application further configures the processor to automatically determine the range of distances corresponding to the restricted depth of field by determining a distance to a surface of a scene object using the set of images captured from different viewpoints.

4. The image processing system of claim 3 , wherein the image processing pipeline application further configures the processor to determine a distance to a surface of a scene object using the set of images captured from different viewpoints by:
generating an initial depth map and a confidence map from at least a portion of the set of images captured from different viewpoints, where the confidence map indicates the reliability of pixel depth estimates in the initial depth map; and
determining the depth of the surface of the scene object based upon at least one pixel depth estimate within the initial depth map marked as confident within the confidence map.

5. The image processing system of claim 3 , wherein the image processing pipeline application further configures the processor to receive a user instruction identifying a surface of a scene object by:
generating a preview image from the set of images captured from different viewpoints, where the preview image includes a user interface cue; and
identifying a surface of a scene object visible within the set of images captured from different viewpoints based upon the location of the user interface cue.

6. The image processing system of claim 3 , wherein the image processing pipeline application further configures the processor to determine the range of distances corresponding to the restricted depth of field based upon user instructions.

7. The image processing system of claim 1 , wherein the image processing pipeline application further configures the processor to generate a restricted depth of field depth map by:
generating an initial depth map using the set of images captured from different viewpoints;
determining pixel locations with depth estimates from the initial depth map indicating that the pixel locations are likely to have depths within the range of distances corresponding to the restricted depth of field;
generating higher depth estimation precision depth estimates for at least some of the pixel locations that are likely to have depths within the range of distances corresponding to the restricted depth of field using the set of images captured from different viewpoints; and
generating a restricted depth of field depth map using at least some of the depth estimates from the initial depth map and at least some of the higher depth estimation precision depth estimates.

8. The image processing system of claim 7 , wherein the image processing pipeline application further configures the processor to generate an initial depth map by:
downsampling at least some of the images in the set of images captured from different viewpoints to obtain a set of lower spatial resolution images; and
determining a low spatial resolution depth map using the set of lower spatial resolution images.

9. The image processing system of claim 8 , wherein the image processing pipeline application further configures the processor to determine a low spatial resolution depth map using the set of lower spatial resolution images by:
performing a disparity search with respect to a given pixel location using the set of lower spatial resolution images;
wherein the disparity search is performed by searching a first set of disparities.

10. The image processing system of claim 9 , wherein the image processing pipeline application further configures the processor to generate the higher precision depth estimates by:
performing a disparity search with respect to a given pixel location using the set of images captured from different viewpoints;
wherein the disparity search is performed by searching a second set of disparities; and
wherein a search performed using the second set of disparities provides greater depth estimation precision within the range of distances corresponding to the restricted depth of field than the precision of a depth estimate obtained within the same range of distances by a search performed using the first set of disparities.

11. The image processing system of claim 10 , wherein the first set of disparities is not uniformly distributed with respect to disparity.

12. The image processing system of claim 10 , wherein the first set of disparities is uniformly distributed with respect to disparity.

13. The image processing system of claim 10 , wherein the second set of disparities is not uniformly distributed with respect to disparity.

14. The image processing system of claim 10 , wherein the second set of disparities is uniformly distributed with respect to disparity.

15. The image processing system of claim 7 , wherein the image processing pipeline application further configures the processor to:
generate an initial confidence map for the initial depth map; and
determine pixel locations with depth estimates from the initial depth map indicating that the pixel locations are likely to have depths within the range of distances corresponding to the restricted depth of field based upon the depth estimate for the pixel location in the initial depth map and the confidence of the depth estimate for the pixel location indicated by the initial confidence map.

16. The image processing system of claim 7 , wherein the image processing pipeline application further configures the processor to determine pixel locations with depth estimates from the initial depth map indicating that the pixel locations are likely to have depths within the range of distances corresponding to the restricted depth of field based upon the depth estimate for the pixel location and a determination that the pixel is not contained within a textureless region.

17. The image processing system of claim 1 , wherein the image processing pipeline application further configures the processor to render a restricted dept

Assignees

• Pelican Imaging Corp
• Kip Peli P1 Lp

Inventors

• Venkataraman Kartik
• Nisenzon Semyon
• Chatterjee Priyam
• Molina Gabriel

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/0065 Primary

  Physics · mapped topic

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title