Depth assisted auto white balance

Patent metadata

Field	Value
Publication number	US-10325354-B2
Application number	US-201715582063-A
Country	US
Kind code	B2
Filing date	Apr 28, 2017
Priority date	Apr 28, 2017
Publication date	Jun 18, 2019
Grant date	Jun 18, 2019

Abstract

A method and apparatus for determining an auto white balance (AWB) gain based on determined depth information. The method may include receiving an image captured by an image sensor, determining depth information associated with the captured image, assigning weights to a plurality of illuminants based on the determined depth information, determining an auto white balance gain based on the assigned weights and statistics of the captured image, and applying the auto white balance gain to the captured image.

First claim

What is claimed is:

1. A method for image processing, the method comprising:
receiving an image captured by an image sensor;
determining depth information associated with the captured image;
assigning weights to a plurality of illuminants based on the determined depth information;
determining an auto white balance gain based on the assigned weights and statistics of the captured image; and
outputting the auto white balance gain.

2. The method of claim 1 , further comprising:
determining a current illuminant based on the determined auto white balance gain, the current illuminant being defined by a correlated color temperature (CCT); and
determining a color correction matrix (CCM) based on the CCT of the current illuminant.

3. The method of claim 1 , wherein detecting the depth information associated with the captured image comprises detecting the depth information at a plurality of locations in the captured image.

4. The method of claim 1 , wherein the statistics of the captured image include a number of near-gray pixels in the image for each of the plurality of illuminants, the method further comprising:
categorizing near-gray pixels in the image into groups associated with respective illuminants of the plurality of illuminants based on respective reference points of the respective illuminants of the plurality of illuminants.

5. The method of claim 1 , further comprising:
determining whether the image was captured indoors or outdoors based on the determined depth information.

6. The method of claim 5 , wherein determining whether the image was captured indoors or outdoors comprises:
determining whether the image was captured indoors or outdoors based on one or more of a maximum depth of the depth information, an average depth of the depth information, or a difference between a minimum and a maximum depth of the depth information.

7. The method of claim 5 , wherein assigning weights to the plurality of illuminants comprises:
assigning relatively higher weights to outdoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured outdoors; and
assigning relatively lower weights to indoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured outdoors.

8. The method of claim 5 , wherein assigning weights to the plurality of illuminants comprises:
assigning relatively lower weights to outdoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured indoors; and
assigning relatively higher weights to indoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured indoors.

9. The method of claim 1 , further comprising:
determining an auto exposure control setting based on the determined depth information.

10. The method of claim 9 , wherein determining the auto exposure control setting comprises:
determining an intended subject of the image based at least in part on the determined depth information; and
determining the auto exposure control setting based on the determined intended subject of the image.

11. An apparatus configured for image processing, the apparatus comprising:
a memory configured to store an image captured by an image sensor; and
one or more processors in communication with the memory, the one or processors configured to:
receive the image;
determine depth information associated with the captured image;
assign weights to a plurality of illuminants based on the determined depth information;
determine an auto white balance gain based on the assigned weights and statistics of the captured image; and
output the auto white balance gain.

12. The apparatus of claim 11 , wherein the one or more processors are further configured to:
determine a current illuminant based on the determined auto white balance gain, the current illuminant being defined by a correlated color temperature (CCT); and
determine a color correction matrix (CCM) based on the CCT of the current illuminant.

13. The apparatus of claim 11 , wherein to detect the depth information associated with the captured image, the one or more processors are further configured to detect the depth information at a plurality of locations in the captured image.

14. The apparatus of claim 11 , wherein the statistics of the captured image include a number of near-gray pixels in the image for each of the plurality of illuminants, and wherein the one or more processors are further configured to:
categorize near-gray pixels in the image into groups associated with respective illuminants of the plurality of illuminants based on respective reference points of the respective illuminants of the plurality of illuminants.

15. The apparatus of claim 11 , wherein the one or more processors are further configured to:
determine whether the image was captured indoors or outdoors based on the determined depth information.

16. The apparatus of claim 15 , wherein to determine whether the image was captured indoors or outdoors, the one or more processors are further configured to:
determine whether the image was captured indoors or outdoors based on one or more of a maximum depth of the depth information, an average depth of the depth information, or a difference between a minimum and a maximum depth of the depth information.

17. The apparatus of claim 15 , wherein to assign weights to the plurality of illuminants, the one or more processors are further configured to:
assign relatively higher weights to outdoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured outdoors; and
assign relatively lower weights to indoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured outdoors.

18. The apparatus of claim 15 , wherein to assign weights to the plurality of illuminants, the one or more processors are further configured to:
assign relatively lower weights to outdoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured indoors; and
assign relatively higher weights to indoor illuminants of the plurality of illuminants in the case that the determined depth information indicates the image was captured indoors.

19. The apparatus of claim 11 , wherein the one or more processors are further configured to:
determine an auto exposure control setting based on the determined depth information.

20. The apparatus of claim 19 , wherein to determine the auto exposure control setting, the one or more processors are further configured to:
determine an intended subject of the image based at least in part on the determined depth information; and
determine the auto exposure control setting based on the determined intended subject of the image.

21. The apparatus of claim 11 , further comprising:
the image sensor configured to capture the image and the depth information.

22. The apparatus of claim 21 , wherein the image sensor comprises one or more of a dual camera system, a dual photodiode sensor, or a time-of-flight camera.

23. A non-transitory computer-readable storage medium storage medium storing instructions that, when executed, cause one or more processors of a device configured for

Assignees

• Qualcomm Inc

Inventors

• Noyes Ying Xie
• Han Kyuseo
• Lee Ho Sang
• Ding Hengzhou
• Chan Leung Chun
• Segapelli Loic Francois

Classifications

• H04N23/88 Primary

  for colour balance, e.g. white-balance circuits or colour temperature control · CPC title

• H04N9/735

  Electricity · mapped topic

• G06T7/50

  Depth or shape recovery · CPC title

• G06T5/00 Primary

  Image enhancement or restoration · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title