Shallow depth of field rendering

Patent metadata

Field	Value
Publication number	US-10410327-B2
Application number	US-201815990154-A
Country	US
Kind code	B2
Filing date	May 25, 2018
Priority date	Jun 2, 2017
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

This disclosure relates to techniques for synthesizing out of focus effects in digital images. Digital single-lens reflex (DSLR) cameras and other cameras having wide aperture lenses typically capture images with a shallow depth of field (SDOF). SDOF photography is often used in portrait photography, since it emphasizes the subject, while deemphasizing the background via blurring. Simulating this kind of blurring using a large depth of field (LDOF) camera may require a large amount of computational resources, i.e., in order to simulate the physical effects of using a wide aperture lens while constructing a synthetic SDOF image. However, cameras having smaller lens apertures, such as mobile phones, may not have the processing power to simulate the spreading of all background light sources in a reasonable amount of time. Thus, described herein are techniques to synthesize out-of-focus background blurring effects in a computationally-efficient manner for images captured by LDOF cameras.

First claim

What is claimed is:

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a first image comprising a first plurality of pixels having values;
obtain a first blur map, wherein the first blur map comprises a plurality of blur values, each blur value corresponding to a pixel in the first plurality of pixels;
for each of one or more pixels of the first plurality of pixels:
select a second plurality of candidate pixels surrounding the respective pixel from the one or more pixels of the first plurality of pixels; and
compute a blurred value for the respective pixel from the one or more pixels of the first plurality of pixels based, at least in part, on a weighted combination of one or more characteristics of each of the second plurality of candidate pixels,
wherein one of the one or more characteristics of each of the second plurality of candidate pixels is based on the respective candidate pixel's corresponding blur value in the first blur map; and

generate a blurred version of the first image based, at least in part, on the computed blur values of the one or more pixels of the first plurality of pixels.

2. The non-transitory program storage device of claim 1 , wherein a candidate pixel's corresponding blur value in the first blur map is used to calculate a degree to which a color or light intensity for the candidate pixel should be included in a given weighted combination.

3. The non-transitory program storage device of claim 1 , wherein the instructions to select a second plurality of candidate pixels further comprise instructions to:
generate a random distribution of pixel locations.

4. The non-transitory program storage device of claim 3 , further comprising instructions to:
reshape the random distribution of pixel locations.

5. The non-transitory program storage device of claim 3 , further comprising instructions to:
correct a non-uniformity of the random distribution of pixel locations.

6. The non-transitory program storage device of claim 1 , wherein one of the one or more characteristics of each of the second plurality of candidate pixels is a value directly proportional to a color or light intensity value of the respective candidate pixel.

7. The non-transitory program storage device of claim 1 , wherein one of the one or more characteristics of each of the second plurality of candidate pixels is a value inversely proportional to the blur value of the respective candidate pixel in the first blur map.

8. The non-transitory program storage device of claim 1 , wherein one of the one or more characteristics of each of the second plurality of candidate pixels is a value inversely proportional to the difference between the blur value of the respective candidate pixel and the blur value of the respective pixel from the one or more pixels of the first plurality of pixels.

9. The non-transitory program storage device of claim 1 , further comprising instructions to:
perform an anti-aliasing operation on at least a portion of the first image after computing the blur values of the one or more pixels of the first plurality of pixels.

10. An imaging device, comprising:
an imaging system;
a programmable control device; and
a memory coupled to the programmable control device, wherein instructions are stored in the memory, and wherein the instructions, when executed, cause the programmable control device to:
obtain, using the imaging system, a first image comprising a first plurality of pixels having values;
obtain a first blur map, wherein the first blur map comprises a plurality of blur values, each blur value corresponding to a pixel in the first plurality of pixels and indicative of the corresponding pixel's distance from a focus plane in the first image;
for each of one or more pixels of the first plurality of pixels:
select a second plurality of candidate pixels surrounding the respective pixel from the one or more pixels of the first plurality of pixels; and
compute a blurred value for the respective pixel from the one or more pixels of the first plurality of pixels based, at least in part, on a weighted combination of one or more characteristics of each of the second plurality of candidate pixels,
wherein one of the one or more characteristics of each of the second plurality of candidate pixels is based on the respective candidate pixel's corresponding blur value in the first blur map; and

generate a blurred version of the first image based, at least in part, on the computed blur values of the one or more pixels of the first plurality of pixels.

11. The imaging device of claim 10 , wherein a candidate pixel's corresponding blur value in the first blur map is used to calculate a degree to which a color or light intensity for the candidate pixel should be included in a given weighted combination.

12. The imaging device of claim 10 , wherein the instructions to select a second plurality of candidate pixels further comprise instructions to:
generate a random distribution of pixel locations.

13. The imaging device of claim 12 , wherein the instructions further comprise instructions to:
reshape the random distribution of pixel locations.

14. The imaging device of claim 12 , wherein the instructions further comprise instructions to:
correct a non-uniformity of the random distribution of pixel locations.

15. The imaging device of claim 10 , wherein one of the one or more characteristics of each of the second plurality of candidate pixels is a value directly proportional to a color or light intensity value of the respective candidate pixel.

16. The imaging device of claim 10 , wherein one of the one or more characteristics of each of the second plurality of candidate pixels is a value inversely proportional to the blur value of the respective candidate pixel in the first blur map.

17. The imaging device of claim 10 , wherein one of the one or more characteristics of each of the second plurality of candidate pixels is a value inversely proportional to the difference between the blur value of the respective candidate pixel and the blur value of the respective pixel from the one or more pixels of the first plurality of pixels.

18. The imaging device of claim 10 , further comprising one or more depth sensors, wherein the blur values are based, at least in part, on information obtained by the one or more depth sensors.

19. An image processing method, comprising:
obtaining a first image comprising a first plurality of pixels having values;
obtaining a first blur map, wherein the first blur map comprises a plurality of blur values, each blur value corresponding to a pixel in the first plurality of pixels;
for each of one or more pixels of the first plurality of pixels:
selecting a second plurality of candidate pixels surrounding the respective pixel from the one or more pixels of the first plurality of pixels; and
computing a blurred value for the respective pixel from the one or more pixels of the first plurality of pixels based, at least in part, on a weighted combination of one or more characteristics of each of the second plurality of candidate pixels,
wherein one of the one or more characteristics of each of the second plurality of candidate pixels is based on the respective candidate pixel's corresponding blur value in the first blur map; and

generating a blurred version of the first image based, at least in part, on the computed blur values of the one or more pixels of the first plurality of pixels.

20. The method o

Assignees

• Apple Inc

Inventors

• Seely Richard D
• Tao Michael W
• Lindskog Alexander
• Anneheim Geoffrey T

Classifications

• H04N23/62

  Control of parameters via user interfaces · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title