Image white balance processing system and method

Patent metadata

Field	Value
Publication number	US-11323676-B2
Application number	US-202016901448-A
Country	US
Kind code	B2
Filing date	Jun 15, 2020
Priority date	Jun 13, 2019
Publication date	May 3, 2022
Grant date	May 3, 2022

Abstract

One limitation of traditional imaging systems is that they are only programmed to correct for a single color of illuminant in a scene. In multi-illuminant scenes, the detected illuminant color may correspond to some mixture of scene illuminants. This may lead to incomplete color correction, wherein, e.g., the dominant illuminant is corrected for but the color cast caused by secondary illuminants is still visible, or an at least partially visible color cast remains from multiple of the scene illuminants. Thus, the techniques disclosed herein comprise: obtaining an image of a scene; generating an illumination map for the obtained image; dividing the values in the illumination map to determine a number of estimated illuminant regions, wherein each region corresponds to at least one estimated illuminant present in the captured scene; estimating a white point for each region; and applying white balancing operations, based on the estimated white points for each region.

First claim

What is claimed is:

1. A device, comprising:
a memory;
one or more image capture devices;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain an image, wherein the image comprises a first plurality of groups of one or more pixels;
generate an illumination map for the image, wherein the illumination map comprises an illuminant estimate for each of the first plurality of groups of one or more pixels;
divide the illuminant estimates from the illumination map into a first plurality of regions, wherein each region in the first plurality of regions corresponds to at least one estimated illuminant that the image was captured under;
estimate a white point for each of the first plurality of regions; and
apply a white balancing gain to each region of the first plurality of regions, wherein the white balancing gain applied to each region corresponds to the respective estimated white point for the region.

2. The device of claim 1 , wherein the first plurality of groups of one or more pixels further comprise a grid of pixel tiles comprising the image, and wherein each pixel tile comprises two or more pixels.

3. The device of claim 1 , wherein the instructions causing the one or more processors to divide the illuminant estimates from the illumination map into a first plurality of regions further comprise instructions causing the one or more processors to:
cluster the illuminant estimates from the illumination map into at most a predetermined maximum number of regions.

4. The device of claim 1 , wherein the instructions causing the one or more processors to divide the illuminant estimates from the illumination map into a first plurality of regions further comprise instructions causing the one or more processors to:
cluster the illuminant estimates from the illumination map using at least one of: a k-means algorithm; or a fuzzy k-means algorithm.

5. The device of claim 1 , wherein the instructions causing the one or more processors to estimate a white point for each of the first plurality of regions further comprise instructions causing the one or more processors to:
constrain each estimated white point towards a Planckian locus or a set of measured artificial light sources.

6. The device of claim 5 , wherein the instructions causing the one or more processors to constrain each estimated white point towards a Planckian locus or a set of measured artificial light sources further comprise instructions causing the one or more processors to:
constrain each estimated white point towards a Planckian locus or a set of measured artificial light sources using a pre-trained matrix.

7. The device of claim 6 , wherein the pre-trained matrix is trained, at least in part, using synthetically-generated image data.

8. The device of claim 1 , wherein the instructions causing the one or more processors to divide the illuminant estimates from the illumination map into a first plurality of regions further comprise instructions causing the one or more processors to divide the illuminant estimates from the illumination map into a first plurality of regions further based, at least in part on:
a scene classification of the image; or
a semantic segmentation operation performed on the image.

9. The device of claim 1 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
smooth a gain map determined based on the estimated white points for the first plurality of regions; and
apply the white balancing gains further based on the smoothed gain map.

10. The device of claim 1 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
estimate a correlated color temperature (CCT) for each of the first plurality of regions; and
apply the white balancing gains further based on the estimated CCTs.

11. The device of claim 1 , wherein the white balancing gains are applied independently to each of the first plurality of regions.

12. A non-transitory computer readable medium comprising computer readable instructions configured to cause one or more processors to:
obtain an image, wherein the image comprises a first plurality of groups of one or more pixels;
generate an illumination map for the image, wherein the illumination map comprises an illuminant estimate for each of the first plurality of groups of one or more pixels;
divide the illuminant estimates from the illumination map into a first plurality of regions, wherein each region in the first plurality of regions corresponds to at least one estimated illuminant that the image was captured under;
estimate a white point for each of the first plurality of regions; and
apply a white balancing gain to each region of the first plurality of regions, wherein the white balancing gain applied to each region corresponds to the respective estimated white point for the region.

13. The non-transitory computer readable medium of claim 12 , wherein the instructions causing the one or more processors to estimate a white point for each of the first plurality of regions further comprise instructions causing the one or more processors to:
constrain each estimated white point towards a Planckian locus or a set of measured artificial light sources.

14. The non-transitory computer readable medium of claim 13 , wherein the instructions causing the one or more processors to constrain each estimated white point towards a Planckian locus or a set of measured artificial light sources further comprise instructions causing the one or more processors to:
constrain each estimated white point towards a Planckian locus or a set of measured artificial light sources using a pre-trained matrix.

15. The non-transitory computer readable medium of claim 12 , wherein a first one of the first plurality of regions corresponds to at least two estimated illuminants.

16. The non-transitory computer readable medium of claim 15 , wherein the instructions causing the one or more processors to estimate a white point for each of the first plurality of regions further comprise instructions causing the one or more processors to estimate a white point for the first one of the first plurality of regions further based, at least in part on:
a weighted combination of an estimated white point for each of the at least two estimated illuminants.

17. An image processing method, comprising:
obtaining an image, wherein the image comprises a first plurality of groups of one or more pixels;
generating an illumination map for the image, wherein the illumination map comprises an illuminant estimate for each of the first plurality of groups of one or more pixels;
dividing the illuminant estimates from the illumination map into a first plurality of regions, wherein each region in the first plurality of regions corresponds to at least one estimated illuminant that the image was captured under;
estimating a white point for each of the first plurality of regions; and
applying a white balancing gain to each region of the first plurality of regions, wherein the white balancing gain applied to each region corresponds to the respective estimated white point for the region.

18. The method of claim 17 , wherein dividing the illuminant estimates from the illumination map into a first plurality of regions further comprises:
clustering the illuminant estimates from the illumination map into at most a predetermined m

Assignees

• Apple Inc

Inventors

• Romanenko Ilya
• Montagna Roberto

Classifications

• H04N23/74 Primary

  by influencing the scene brightness using illuminating means · CPC title

• H04N25/134 Primary

  based on three different wavelength filter elements · CPC title

• H04N23/88

  for colour balance, e.g. white-balance circuits or colour temperature control · CPC title

• H04N5/2354

  Electricity · mapped topic

• H04N9/73 Primary

  Colour balance circuits, e.g. white balance circuits or colour temperature control (camera processing pipelines for colour balance H04N23/88) · CPC title