Hue map generation for highlight recovery

Patent metadata

Field	Value
Publication number	US-10855964-B2
Application number	US-201816116838-A
Country	US
Kind code	B2
Filing date	Aug 29, 2018
Priority date	Aug 29, 2018
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

Embodiments relate to generation of hue maps for highlight recovery of an input image. An image having a plurality of color channels is obtained at a first resolution lower than a resolution of the input image. A hue for each color channel for each pixel is determined, using a pixel value for that color channel and pixel values for the plurality of color channels in the first image. Weights are determined for each pixel for each color channel, based on hues for the pixel and pixel values for the pixel in the first image. A plurality of candidate hue maps are generated, based on the weights and pixel values in the first image in a patch surrounding the pixel for the plurality of color channels.

First claim

What is claimed is:

1. An apparatus for processing an input image, comprising:
a filter circuit configured to:
determine filter coefficients for each pixel in a first image at a first resolution lower than a resolution of the input image and captured within a threshold time interval from capturing the input image, using differences between pixel values of the first image and other pixel values of a second image for each color channel of a plurality of color channels in a patch surrounding each pixel, the second image at the first resolution having the plurality of color channels, and
filter the pixel values of the first image for each color channel using the filter coefficients for each pixel to generate a filtered image having the plurality of color channels; and

a hue map generator circuit configured to:
determine a hue for each color channel for each pixel in the filtered image, using a corresponding pixel value for each color channel and corresponding pixel values for the plurality of color channels in the filtered image,
determine a plurality of weights for the plurality of color channels for each pixel in the filtered image, based on hues for each pixel for the plurality of color channels and the corresponding pixel values, and
generate a plurality of candidate hue maps at the first resolution for the plurality of color channels, based on the weights for each pixel and pixel values in the filtered image in a patch surrounding each pixel for the plurality of color channels.

2. The apparatus of claim 1 , further comprising:
a hue target circuit configured to generate a target hue for a pixel in the input image, based on the plurality of candidate hue maps; and
a hue recovery circuit to generate a recovered version of the input image by adjusting hue information of the pixel in the input image based on the generated target hue, the input image comprising the pixel with the hue information having a clipped color channel.

3. The apparatus of claim 1 , further comprising:
a front-end hue processor configured to process a plurality of pixel values for each color channel in the first image to generate a first candidate hue map at the first resolution and having the plurality of color channels.

4. The apparatus of claim 3 , wherein the front-end hue processor is further configured to:
apply at least one of black level compensation and lens shading correction on the pixel values for each color channel in the first image.

5. The apparatus of claim 3 , further comprising a white balance correction circuit configured to:
correct one or more pixel values for each color channel in the first candidate hue map based on a white balance gain for each color channel to generate a corrected version of the first candidate hue map.

6. The apparatus of claim 1 , further comprising a blending circuit configured to:
determine a first blending weight for each pixel in the filtered image, based on the filter coefficients for each pixel in the first image,
determine a second blending weight for each pixel in the filtered image, based on differences between pixel values of the filtered image and the second image and on differences between pixel values of the first and second images for the plurality of color channels,
determine a blending weight for each pixel in the filtered image, based on the first blending weight and the second blending weight, and
generate a motion compensated image having the plurality of color channels, using the blending weight for each pixel in the filtered image, the filtered image and the first image.

7. The apparatus of claim 6 , further comprising a white balance correction circuit configured to:
correct one or more pixel values for each color channel in the motion compensated image based on a white balance gain for each color channel.

8. The apparatus of claim 1 , wherein the hue map generator circuit is further configured to:
determine, for each pixel in the filtered image, a color factor for each color channel, based on hues for each pixel in the filtered image for the plurality of color channels, and
determine, for each pixel in the filtered image, a weight of the plurality of weights for each color channel, based on the color factor and a pixel value in the filtered image for a corresponding color channel of the plurality of color channels.

9. The apparatus of claim 1 , wherein the hue map generator circuit is further configured to:
determine for each color channel a gradient for each pixel in the filtered image, using corresponding pixel values in a candidate hue map of the plurality of candidate hue maps for each color channel, and
generate a modified hue map for each color channel, based on a corresponding candidate hue map of the plurality of candidate hue maps for each color channel and the gradient for each pixel for each color channel.

10. A method, comprising:
determining filter coefficients for each pixel in a first image at a first resolution lower than a resolution of the input image and captured within a threshold time interval from capturing the input image, using differences between pixel values of the first image and other pixel values of a second image for each color channel of a plurality of color channels in a patch surrounding each pixel, the second image at the first resolution having the plurality of color channels;
filtering the pixel values of the first image for each color channel using the filter coefficients for each pixel to generate a filtered image having the plurality of color channels;
determining a hue for each color channel for each pixel in the filtered image, using a corresponding pixel value for each color channel and corresponding pixel values for the plurality of color channels in the filtered image;
determining a plurality of weights for the plurality of color channels for each pixel in the filtered image, based on hues for each pixel for the plurality of color channels and the corresponding pixel values; and
generating a plurality of candidate hue maps at the first resolution for the plurality of color channels, based on the weights for each pixel and pixel values in the filtered image in a patch surrounding each pixel for the plurality of color channels.

11. The method of claim 10 , further comprising:
generating a target hue for a pixel in the input image, based on the plurality of candidate hue maps; and
generating a recovered version of the input image by adjusting hue information of the pixel in the input image based on the generated target hue, the input image comprising the pixel with the hue information having a clipped color channel.

12. The method of claim 10 , further comprising:
processing a plurality of pixel values for each color channel in the first image to generate a first candidate hue map at the first resolution and having the plurality of color channels.

13. The method of claim 12 , further comprising:
applying at least one of black level compensation and lens shading correction on the pixel values for each color channel in the first image.

14. The method of claim 10 , further comprising:
determining a first blending weight for each pixel in the filtered image, based on the filter coefficients for each pixel in the first image;
determining a second blending weight for each pixel in the filtered image, based on differences between pixel values of the filtered image and the second image and on differences between pixel values of the first and second images for the plurality of color channels;
determining a blending weight for each pixel in the filtered image, based on the first blending weight and the second blending weigh

Assignees

• Apple Inc

Inventors

• Cao Frederic
• Tajbakhsh Touraj
• Wang Muge

Classifications

• H04N23/684

  performed by controlling the image sensor readout, e.g. by controlling the integration time · CPC title

• H04N23/88

  for colour balance, e.g. white-balance circuits or colour temperature control · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• H04N9/643 Primary

  Hue control means, e.g. flesh tone control · CPC title