System for reducing depth of field with digital image processing

Patent metadata

Field	Value
Publication number	US-9230306-B2
Application number	US-201313760569-A
Country	US
Kind code	B2
Filing date	Feb 6, 2013
Priority date	Feb 7, 2012
Publication date	Jan 5, 2016
Grant date	Jan 5, 2016

Abstract

An electronic device may have a camera module. The camera module may capture images having an initial depth of field. The electronic device may receive user input selecting a focal plane and an effective f-stop for use in producing a modified image with a reduced depth of field. The electronic device may include image processing circuitry that selectively blurs various regions of a captured image, with each region being blurred to an amount that varies with distance to the user selected focal plane and in response to the user selected effective f-stop (e.g., a user selected level of depth of field).

First claim

What is claimed is:

1. A method of defocusing a first image to produce a second image, the method comprising:
obtaining the first image and a depth map associated with the first image;
for each of a plurality of pixels of the first image and using the depth map, determining the distance between that pixel and a focal plane in the second image;
for each of the plurality of pixels of the first image, identifying an associated sampling region that includes that pixel and that has an area at least based on the distance between that pixel and the focal plane; and
for each of the plurality of pixels of the first image, identifying a given number of sample pixels that includes at least that pixel and each of which lie within the associated sampling region of that pixel, wherein the given number of sample pixels is substantially equal for all of the sample regions.

2. The method defined in claim 1 further comprising:
for each of the plurality of pixels of the first image, averaging together image values of all of the sample pixels that lie within the associated sampling region of that pixel to generated a defocused pixel value; and
producing the second image using the defocused pixel values.

3. The method defined in claim 1 wherein all of the sample regions have an area that is greater than a given threshold area, wherein the plurality of pixels of the first image comprises a first plurality of pixels of the first image, and wherein the method further comprises:
for each of a second plurality of pixels of the first image and using the depth map, determining the distance between that pixel and the focal plane in the second image;
for each of the second plurality of pixels of the first image, identifying an associated sampling region that includes that pixel and that has an area at least based on the distance between that pixel and the focal plane; and
for each of the second plurality of pixels of the first image, identifying at least one sample pixel that includes at least that pixel and each of which lies within the associated sampling region of that pixel, wherein the number of sampling pixels identified for each of the second plurality of pixels is less than the given number of sample pixels.

4. The method defined in claim 3 wherein each of the sampling regions associated with the second plurality of pixels has an area that is less than the given threshold area.

5. The method defined in claim 4 wherein, for each of the first plurality of pixels, the distance between that pixel and the focal plane is greater than a given distance and wherein, for each of the second plurality of pixels, the distance between that pixel and the focal plane is less than the given distance.

6. The method defined in claim 1 wherein each of the plurality of pixels of the first image has at a respective depth in the depth map and wherein identifying the sample pixels for each of the plurality of pixels comprises:
for each of the plurality of pixels, setting a back threshold equal to the depth of that pixel plus a threshold distance; and
for each of the plurality of pixels, selecting, as sample pixels, pixels that have a depth in the depth map that is less than or equal to the back threshold of that pixel and rejecting, as sample pixels, any pixels that have a depth in the depth map that is greater than the back threshold of that pixel.

7. The method defined in claim 1 wherein the first image comprises a stereo image formed from left and right images.

8. The method defined in claim 7 wherein obtaining the depth map comprises determining stereo disparity for the plurality of pixels of the first image based on stereo differences between the left and right image.

9. The method defined in claim 8 wherein the area of each of the sampling regions is linearly proportional to the stereo disparity of the associated pixel.

10. A method, comprising:
obtaining a first image having a plurality of first pixels, wherein the first image has a first depth of field and is substantially in focus between first and second depths;
obtaining a depth value for each of the first pixels; and
using the depth values, defocusing the first image to create a second image that has a second depth of field and that is substantially in focus between third and fourth depths, wherein defocusing the first image to create the second image comprises:
calculating a plurality of defocused pixel values for the second image, wherein each defocused pixel value is calculated using a given number of pixels from the first image and wherein the given number of pixels used in calculating each defocused pixel value is substantially equal for each of the defocused pixel values.

11. The method defined in claim 10 wherein the second depth of field is less than the first depth of field and wherein the third and fourth depths each lie between the first and second depths.

12. The method defined in claim 11 wherein the second image has a focal plane located approximately midway between the third and fourth depths, wherein calculating each of the defocused pixel values comprises:
starting with an initial pixel in the first image, determining the distance between the initial pixel and the focal plane of the second image; and
identifying a sampling region approximately centered on the initial pixel, wherein the sampling region has a width that is proportional to the distance between the initial pixel and the focal plane.

13. The method defined in claim 12 further comprising:
determining the location of the focal plane of the second image in response to user input identifying the location of the focal plane.

14. The method defined in claim 13 wherein there is a difference between the third and fourth depths, over which the second image is substantially in focus, the further comprising:
determining the magnitude of the difference between the third and fourth depths in response to user input identifying the second depth of field of the second image.

15. The method defined in claim 14 wherein the widths of the sampling regions are at least partly based on the user input identifying the second depth of field of the second image.

16. A system, comprising:
a central processing unit;
memory;
input-output circuitry;
an imaging device that captures a first image having a first depth of field;
image processing circuitry that defocuses the first image to create a second image having a second depth of field that is less than the first depth of field, wherein the image processing circuitry defocuses the first image using a plurality of defocus kernels each formed from a given number of pixel samples from the first image, wherein each of the defocus kernels has a width that is proportional to its distance from a focal plane in the second image and wherein the given number of pixel samples in each of the defocus kernels is substantially independent of the widths of the defocus kernels.

17. The system defined in claim 16 further comprising:
a touch screen that displays the first image, that receives user input comprising a touch event on a portion of the displayed first image that identifies the focal plane in the second image, and that displays the second image after receiving the user input.

Assignees

• Semiconductor Components Ind

Inventors

• Sun Yajie
• Pope David R

Classifications

• H04N13/246

  Calibration of cameras · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N5/2621

  Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability · CPC title

• H04N13/0246

  Electricity · mapped topic

• G06T5/002

  Physics · mapped topic