Saturated pixel recovery in light-field images

Patent metadata

Field	Value
Publication number	US-9635332-B2
Application number	US-201414480340-A
Country	US
Kind code	B2
Filing date	Sep 8, 2014
Priority date	Sep 8, 2014
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

According to various embodiments, the system and method disclosed herein process light-field image data so as to mitigate or remove the effects of pixel saturation in light-field images. A light-field image may be captured with a light-field image capture device with a microlens array. The light-field image may be demodulated, and then a recovered value for each saturated pixel of the demodulated light-field image may be obtained. This may be done by comparing a saturated value of each saturated pixel with reference values of reference pixels proximate the saturated pixel and, if the proximate pixels have a higher reference value than the saturated value, setting the saturated pixel to the reference value. This process may be carried out iteratively until recovered values are obtained for the saturated pixels. A demodulated, saturation-recovered light-field image may then be generated and displayed with the recovered values for the saturated pixels.

First claim

What is claimed is:

1. In a light-field image capture device having a sensor and a plurality of microlenses, a method for compensating for sensor saturation and microlens modulation, the method comprising:
in a processor, receiving a light-field image;
in the processor, generating a demodulated light-field image based on the light-field image;
in the processor, obtaining a first recovered value for a first saturated pixel of the demodulated light-field image, comprising:
determining that at least a first reference pixel proximate the first saturated pixel comprises a first reference value higher than a first saturated value of the first saturated pixel; and
based on the determination, establishing a first recovered value for the first saturated pixel, wherein the first recovered value is higher than the first saturated value;

in the processor, generating a demodulated, saturation-recovered light field image in which the first saturated pixel has the first recovered value; and
on a display device, displaying the demodulated, saturation-recovered light-field image.

2. The method of claim 1 , wherein generating the demodulated light-field image comprises:
in the processor, determining a flat-field response contour for each of at least one region of an image sensor;
in the processor, generating a modulation image based on the at least one flat-field response contour; and
in the processor, generating a demodulation image from the modulation image.

3. The method of claim 1 , further comprising, prior to generation of the demodulated light-field image, identifying the first saturated pixel by determining that a pre-demodulation value of the saturated pixel value is equal to a maximum value that can be captured the sensor.

4. The method of claim 1 , wherein the light-field image comprises a RAW light-field image comprising a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, the method comprising:
applying a demosaicing operation to a selection from the group consisting of the light-field image, the demodulated light-field image, and the saturation-recovered light-field image.

5. The method of claim 4 , wherein applying the demosaicing operation comprises applying the demosaicing operation on the saturation-recovered light-field image.

6. The method of claim 4 , wherein applying the demosaicing operation comprises applying the demosaicing operation prior to obtaining the first recovered value.

7. The method of claim 4 , wherein the plurality of pixels are arranged in a Bayer pattern.

8. The method of claim 1 , wherein obtaining the first recovered value for the first saturated pixel of the demodulated light-field image further comprises, prior to determining that the first reference value is higher than the first saturated value:
iteratively comparing an intermediate saturated value of the first saturated pixel with an intermediate reference value of the first reference pixel; and
in each iteration in which the intermediate reference value is greater than the intermediate saturated value, establishing the intermediate reference value as a new intermediate saturated value of the saturated pixel until the new intermediate saturated value is the first saturated value.

9. The method of claim 1 , wherein obtaining the first recovered value for the first saturated pixel of the demodulated light-field image further comprises:
calculating an average reference value of a plurality of pixels proximate the first saturated pixel;
wherein determining that at least one additional reference pixel proximate the first saturated pixel comprises at least one additional reference value higher than the first saturated value comprises determining that the average reference value is higher than the first saturated value;
and wherein establishing the first recovered value comprises setting the first recovered value equal to the average reference value.

10. The method of claim 9 , wherein computing the average reference value of the plurality of pixels proximate the first saturated pixel comprises computing the average reference value of four pixels that are displaced from the first saturated pixel along four directions that are generally orthogonal to each other.

11. The method of claim 1 , wherein the first saturated pixel is located in a first saturated pixel region of the demodulated light-field image, wherein the first saturated pixel region comprises, in addition to the first saturated pixel, a plurality of additional saturated pixels, the method further comprising:
obtaining an additional recovered value for each pixel of the plurality of additional saturated pixels;
wherein, in the saturation-recovered light-field image, each of the additional saturated pixels has the corresponding additional recovered value.

12. The method of claim 11 , wherein the demodulated light-field image further comprises, in addition to the first saturated pixel region, a plurality of additional saturated pixel regions, wherein each of the first saturated pixel region and the additional saturated pixel regions comprises light-field data received through a microlens of the plurality of microlenses;
wherein generating the demodulated light-field image comprises, for each of the first saturated pixel region and the additional saturated pixel regions, creating a region of artificially low pixel values within the demodulated light-field image.

13. The method of claim 1 , wherein receiving the light-field image comprises capturing the light-field image with the light-field image capture device.

14. A computer program product for compensating for sensor saturation and microlens modulation in a light-field image capture device having a sensor and a plurality of microlenses, the computer program product comprising:
a non-transitory storage medium; and
computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of:
receiving a light-field image;
generating a demodulated light-field image based on the light-field image;
obtaining a first recovered value for a first saturated pixel of the demodulated light-field image, comprising:
determining that at least a first reference pixel proximate the first saturated pixel comprises a first reference value higher than a first saturated value of the first saturated pixel; and
based on the determination, establishing a first recovered value for the first saturated pixel, wherein the first recovered value is higher than the first saturated value;

generating a demodulated, saturation-recovered light field image in which the first saturated pixel has the first recovered value; and
causing a display device to display the demodulated, saturation-recovered light-field image.

15. The computer program product of claim 14 , wherein the computer program code configured to cause the at least one processor to generate the demodulated light-field image comprises computer program code configured to cause the at least one processor to perform the steps of:
determining a flat-field response contour for each of at least one region of an image sensor;
generating a modulation image based on the at least one flat-field response contour; and
generating a demodulation image from the modulation image.

16. The computer program product of claim 14 , wherein the light-field image comprises a RAW light-field image comprising a plurality of pixels, each pixel being associated with a plurality of values, each value corresponding to a different spectral range, wherein the co

Assignees

• Lytro Inc

Inventors

• Carroll Robert

Classifications

• H04N23/81 Primary

  for suppressing or minimising disturbance in the image signal generation · CPC title

• H04N23/88 Primary

  for colour balance, e.g. white-balance circuits or colour temperature control · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title

• H04N23/843

  Demosaicing, e.g. interpolating colour pixel values · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title