3D color mapping and tuning in an image processing pipeline

Patent metadata

Field	Value
Publication number	US-9961236-B2
Application number	US-201615181143-A
Country	US
Kind code	B2
Filing date	Jun 13, 2016
Priority date	Jun 13, 2016
Publication date	May 1, 2018
Grant date	May 1, 2018

Abstract

The processing of RGB image data can be optimized by performing optimization operations on the image data when it is converted into the YCbCr color space. First, a raw RGB color space is converted into a YCbCr color space, and raw RGB image data is converted into YCbCr image data using the YCbCr color space. For each Y-layer of the YCbCr image data, a 2D LUT is generated. The YCbCr image data is converted into optimized CbCr image data using the 2D LUTs, and optimized YCbCr image data is generated by blending CbCr image data corresponding to multiple Y-layers. The optimized YCbCr image data is converted into sRGB image data, and a tone curve is applied to the sRGB image data to produce optimized sRGB image data.

First claim

What is claimed is:

1. A camera system, comprising:
an image sensor configured to convert light incident upon the image sensor into RGB image data;
an image signal processor (“ISP”) configured to:
access a set of linear 3D color points corresponding to an input color space;
generate a color-mapped linear 3D color look-up table (“LUT”) based on the set of linear 3D color points;
generate a gain curve based on the RGB image data;
apply the gain curve to the color mapped linear 3D color look-up table to generate a tone-mapped color-mapped linear 3D color LUT; and
apply the tone-mapped color-mapped linear 3D color LUT to the RGB image data to generate tone-mapped and color-mapped output image data; and

a memory configured to store the tone-mapped color-mapped linear 3D color LUT and the tone-mapped and color-mapped output image data.

2. The camera system of claim 1 , wherein applying the gain curve to the color-mapped linear 3D color LUT comprises:
determining a luminance intensity of each color in the color-mapped linear 3D color LUT; and
applying the luminance intensities to the color-mapped linear 3D color LUT.

3. The camera system of claim 1 , wherein generating a gain curve based on the RGB image data comprises generating a gain curve based on a downscaled version of the RGB image data.

4. The camera system of claim 1 , wherein the ISP is further configured to:
apply an encoding gamma curve to the tone-mapped color-mapped linear 3D color LUT to produce a tone-mapped color-mapped nonlinear 3D color LUT.

5. The camera system of claim 1 , wherein generating a color-mapped linear 3D color LUT comprises:
applying a first 3D color LUT with a fixed encoding gamma curve to the set of linear 3D color points to generate a set of non-linear color-mapped color points.

6. The camera system of claim 5 , wherein generating a color-mapped linear 3D color LUT further comprises:
applying a decoding gamma curve to the non-linear color-mapped color points to generate the color-mapped linear 3D color points.

7. The camera system of claim 1 , wherein generating a gain curve based on the RGB image data comprises:
applying a first 3D color LUT with a fixed encoding gamma curve to the RGB image data to produce nonlinear color-mapped color points; and
applying a decoding gamma curve to the nonlinear color-mapped color points to produce color-mapped linear RGB image data.

8. The camera system of claim 7 , wherein generating a gain curve based on the RGB image data further comprises:
applying a global tone mapping to the color-mapped linear RGB image data to produce the gain curve.

9. The camera system of claim 1 , wherein the RGB image data comprises a plurality of pixel values that, respectively, correspond to a plurality of pixels of an image.

10. The camera system of claim 9 , wherein applying the tone-mapped color-mapped linear 3D color LUT to the RGB image data includes determining output pixel values for individual ones of the plurality of pixels of the image based on the tone-mapped color-mapped linear 3D color LUT and the plurality of pixel values of the RBG image data.

11. A method, comprising:
converting, by a camera, light incident upon an image sensor of the camera into raw RGB image data;
accessing, by an image signal processor (“ISP”) of the camera, a set of linear 3D color points corresponding to an input color space;
generating, by the ISP of the camera, a color-mapped linear 3D color look-up table (“LUT”) based on the set of linear 3D color points;
generating, by the ISP of the camera, a gain curve based on the RGB image data;
applying, by the ISP of the camera, the gain curve to the color mapped linear 3D color look-up table to generate a tone-mapped color-mapped linear 3D color LUT; and
applying, by the ISP of the camera, the tone-mapped color-mapped linear 3D color LUT to the RGB image data to generate tone-mapped and color-mapped output image data; and
storing, by a memory of the camera, the tone-mapped color-mapped output image.

12. The method of claim 11 , wherein applying the gain curve to the color-mapped linear 3D color LUT comprises:
determining a luminance intensity of each color in the color-mapped linear 3D color LUT; and
applying the luminance intensities to the color-mapped linear 3D color LUT.

13. The method of claim 11 , wherein generating a gain curve based on the RGB image data comprises generating a gain curve based on a downscaled version of the RGB image data.

14. The method of claim 11 , further comprising:
applying, by the ISP of the camera, an encoding gamma curve to the tone-mapped color-mapped linear 3D color LUT to produce a tone-mapped color-mapped nonlinear 3D color LUT.

15. The method of claim 11 , wherein generating a color-mapped linear 3D color LUT comprises:
applying a first 3D color LUT with a fixed encoding gamma curve to the set of linear 3D color points to generate a set of non-linear color-mapped color points.

16. The method of claim 15 , wherein generating a color-mapped linear 3D color LUT further comprises:
applying a decoding gamma curve to the non-linear color-mapped color points to generate the color-mapped linear 3D color points.

17. The method of claim 11 , wherein generating a gain curve based on the RGB image data comprises:
applying a first 3D color LUT with a fixed encoding gamma curve to the RGB image data to produce nonlinear color-mapped color points; and
applying a decoding gamma curve to the nonlinear color-mapped color points to produce color-mapped linear RGB image data.

18. The method of claim 17 , wherein generating a gain curve based on the RGB image data further comprises:
applying a global tone mapping to the color-mapped linear RGB image data to produce the gain curve.

19. The method of claim 11 , wherein the RGB image data comprises a plurality of pixel values that, respectively, correspond to a plurality of pixels of an image.

20. The method of claim 19 , wherein applying the tone-mapped color-mapped linear 3D color LUT to the RGB image data includes determining output pixel values for individual ones of the plurality of pixels of the image based on the tone-mapped color-mapped linear 3D color LUT and the plurality of pixel values of the RBG image data.

Assignees

• Gopro Inc

Inventors

• Chen Ying
• Chawla Sumit
• Edwards Elizabeth Torres
• Chandu Kartheek
• Ding Bo

Classifications

• H04N1/6008

  with primary colour signals, e.g. RGB or CMY(K) · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• H04N1/6005

  with luminance or chrominance signals, e.g. LC1C2, HSL or YUV · CPC title

• H04N1/6019 Primary

  using look-up tables (H04N1/6025 takes precedence) · CPC title

• H04N23/86

  for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits · CPC title