Image capture for later refocusing or focus-manipulation

Patent metadata

Field	Value
Publication number	US-9501834-B2
Application number	US-201213346602-A
Country	US
Kind code	B2
Filing date	Jan 9, 2012
Priority date	Aug 18, 2011
Publication date	Nov 22, 2016
Grant date	Nov 22, 2016

Abstract

A system, method, and computer program product for capturing images for later refocusing. Embodiments estimate a distance map for a scene, determine a number of principal depths, capture a set of images, with each image focused at one of the principal depths, and process captured images to produce an output image. The scene is divided into regions, and the depth map represents region depths corresponding to a particular focus step. Entries having a specific focus step value are placed into a histogram, and depths having the most entries are selected as the principal depths. Embodiments may also identify scene areas having important objects and include different important object depths in the principal depths. Captured images may be selected according to user input, aligned, and then combined using blending functions that favor only scene regions that are focused in particular captured images.

First claim

What is claimed is:

1. A processor-implemented method for capturing images for later refocusing, comprising:
estimating a depth map for a scene;
determining a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;
capturing a set of images, each image focused at one of the principal depths; and
processing captured images to produce an output image.

2. The method of claim 1 , wherein the estimating comprises depth from defocus.

3. The method of claim 1 , wherein the estimating comprises acquiring images at different focus settings and finding the focus setting yielding a maximum spatial derivative for at least a portion of the scene.

4. The method of claim 1 , wherein the estimating comprises employing an external rangefinder.

5. The method of claim 1 , wherein the determining further comprises:
dividing the scene into a plurality of rectangles, wherein the depth map represents depths of the rectangles, the depths corresponding to one of a predetermined number of focus steps; and
selecting depths having the most depth map entries as the principal depths.

6. The method of claim 1 , wherein the determining further comprises:
dividing the scene into a plurality of rectangles, wherein the depth map represents depths of the rectangles, the depths corresponding to one of a predetermined number of focus steps;
counting depth map entries having a specific focus step value;
sorting counted depth map entries into a histogram; and
selecting depths having the most depth map entries as the principal depths.

7. The method of claim 6 , wherein the number of principal depths is a fraction of the number of focus steps, the fraction calculated according to a minimum number of focus steps from the histogram required to cover a predetermined percentage of the rectangles.

8. The method of claim 1 , wherein the determining further comprises:
identifying scene areas having important objects; and
including different important object depths in the principal depths.

9. The method of claim 8 , wherein the important objects include at least one of faces, pets, cars, people, and recognized faces of specific persons.

10. The method of claim 8 , wherein the important object depths are placed first in the principal depths.

11. The method of claim 10 , further comprising, if the number of important objects is less than a predetermined number, adding depths corresponding to highest histogram counts but not corresponding to important objects to the principal depths.

12. The method of claim 8 , further comprising, if the number of important objects is at least a predetermined number, the number of important objects determines the number of principal depths.

13. The method of claim 8 , wherein the number of principal depths is limited by at least one of available memory and the number of images that can be captured substantially simultaneously.

14. The method of claim 1 , wherein the processing comprises selecting one image from the set according to at least one of user input and automated selection criteria.

15. The method of claim 14 , wherein the selection criteria include at least one of which image has a recognized face in focus, and which image has the highest depth map histogram count.

16. The method of claim 1 , wherein the processing comprises:
selecting a plurality of images from the set according to at least one of user input and automated selection criteria;
aligning the selected images; and
combining the selected images using blending functions.

17. The method of claim 16 , wherein each blending function equals one in areas of a corresponding selected image that are focused, zero in areas of other selected images that are focused, and values between zero and one in other areas.

18. The method of claim 16 , wherein the plurality includes all the captured images.

19. The method of claim 1 , further comprising generating an output video clip based on at least part of the processed images.

20. A computer program product comprising a machine-readable medium tangibly embodying non-transitory program instructions thereon that, when executed by the machine, cause the machine to capture images for later refocusing by:
estimating a depth map for a scene;
determining a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;
capturing a set of images, each image focused at one of the principal depths; and
processing captured images to produce an output image.

21. A system for capturing images for later refocusing, comprising:
means for estimating a depth map for a scene;
means for determining a number of principal depths, wherein the means for determining comprises means for selecting a plurality of depths having the largest areas in the depth map as the principal depths;
means for capturing a set of images, each image focused at one of the principal depths; and
means for processing captured images to produce an output image.

22. A system for capturing images for later refocusing, comprising:
a processor; and
a memory that stores instructions, the processor instruction execution causing the processor to:
estimate a depth map for a scene;
determine a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;
capture a set of images, each image focused at one of the principal depths; and
process captured images to produce an output image.

23. An integrated circuit for capturing images for later refocusing, comprising:
a first circuit element for estimating a depth map for a scene;
a second circuit element for determining a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;
a third circuit element for capturing a set of images, each image focused at one of the principal depths; and
a fourth circuit element for processing captured images to produce an output image.

Assignees

• Tzur Meir
• Qualcomm Technologies Inc

Inventors

• Tzur Meir

Classifications

• G02B27/0075 Primary

  with means for altering, e.g. increasing, the depth of field or depth of focus · CPC title

• H04N23/743 Primary

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T7/0069 Primary

  Physics · mapped topic