Tone mapping in high-resolution imaging systems

Patent metadata

Field	Value
Publication number	US-2024257325-A1
Application number	US-202318485970-A
Country	US
Kind code	A1
Filing date	Oct 12, 2023
Priority date	Jan 26, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

A method includes obtaining multiple image frames captured using at least one imaging sensor. The method also includes generating a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor. The method further includes generating a blended and demosaiced image based on the image frames and generating a local tone mapped image based on the blended and demosaiced image and the local tone map. The method also includes adjusting color saturation based on the local tone mapped image to generate a corrected image. In addition, the method includes generating an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

First claim

What is claimed is:

1 . A method comprising:
obtaining multiple image frames captured using at least one imaging sensor; generating a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor; generating a blended and demosaiced image based on the image frames; generating a local tone mapped image based on the blended and demosaiced image and the local tone map; adjusting color saturation based on the local tone mapped image to generate a corrected image; and generating an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

2 . The method of claim 1 , wherein generating the local tone map comprises:
selecting one of the image frames; generating a gain map based on the selected image frame; dividing the gain map into multiple tiles; and generating a three-dimensional (3D) LUT using the tiles.

3 . The method of claim 2 , wherein generating the gain map comprises:
down-sampling the selected image frame to generate a down-sampled image frame; converting the down-sampled image frame into a single luma channel image; applying local tone mapping to the single luma channel image to generate a tone mapped image; and determining gain values of the gain map based on ratios of values in the tone mapped image and values in the single luma channel image.

4 . The method of claim 3 , wherein generating the one or more contrast enhancement LUTs comprises generating the one or more contrast enhancement LUTs based on the tone mapped image.

5 . The method of claim 2 , wherein generating the 3D LUT comprises:
determining gain values for locations of a profile gain table map (PGTM); filling in one or more gaps of the PGTM, each gap associated with a gain value that is initially missing in the PGTM due to a lack of image data, the PGTM defining intensity-gain curves; for one or more of the intensity-gain curves that end prematurely, concatenating each of the one or more intensity-gain curves with a parameterized tail; filtering the intensity-gain curves in an intensity dimension and in a spatial dimension; and for at least one of the intensity-gain curves having a parameterized tail with a gradient larger than a threshold, smoothing the parameterized tail.

6 . The method of claim 1 , wherein generating the global tone map LUT comprises generating the global tone map LUT based on the one or more parameters of the at least one imaging sensor.

7 . The method of claim 1 , wherein generating the local tone mapped image comprises:
applying the local tone map to the blended and demosaiced image to generate a first gain map; performing a morphological opening operation using the first gain map to generate a second gain map; performing a guided filtering operation using the first and second gain maps to generate a third gain map; and generating the local tone mapped image using the blended and demosaiced image and the third gain map.

8 . The method of claim 1 , wherein:
the color saturation is adjusted prior to performing a gamma correction; the corrected image undergoes the gamma correction and is converted from a red-green-blue (RGB) image domain to a luma-chroma (YUV) image domain; and generating the output image comprises applying contrast enhancement using the one or more contrast enhancement LUTs and applying global tone mapping using the global tone map LUT to a luma channel in the YUV image domain.

9 . An electronic device comprising:
at least one imaging sensor configured to capture multiple image frames; and at least one processing device configured to:
generate a local tone map, a global tone map look-up table (LUT), and one or more contrast enhancement LUTs based on at least one of the image frames and one or more parameters of the at least one imaging sensor;
generate a blended and demosaiced image based on the image frames;
generate a local tone mapped image based on the blended and demosaiced image and the local tone map;
adjust color saturation based on the local tone mapped image to generate a corrected image; and
generate an output image based on the corrected image, the global tone map LUT, and the one or more contrast enhancement LUTs.

10 . The electronic device of claim 9 , wherein, to generate the local tone map, the at least one processing device is configured to:
select one of the image frames; generate a gain map based on the selected image frame; divide the gain map into multiple tiles; and generate a three-dimensional (3D) LUT using the tiles.

11 . The electronic device of claim 10 , wherein, to generate the gain map, the at least one processing device is configured to:
down-sample the selected image frame to generate a down-sampled image frame; convert the down-sampled image frame into a single luma channel image; apply local tone mapping to the single luma channel image to generate a tone mapped image; and determine gain values of the gain map based on ratios of values in the tone mapped image and values in the single luma channel image.

12 . The electronic device of claim 11 , wherein the at least one processing device is configured to generate the one or more contrast enhancement LUTs based on the tone mapped image.

13 . The electronic device of claim 10 , wherein, to generate the 3D LUT, the at least one processing device is configured to:
determine gain values for locations of a profile gain table map (PGTM); fill in one or more gaps of the PGTM, each gap associated with a gain value that is initially missing in the PGTM due to a lack of image data, the PGTM defining intensity-gain curves; for one or more of the intensity-gain curves that end prematurely, concatenate each of the one or more intensity-gain curves with a parameterized tail; filter the intensity-gain curves in an intensity dimension and in a spatial dimension; and for at least one of the intensity-gain curves having a parameterized tail with a gradient larger than a threshold, smooth the parameterized tail.

14 . The electronic device of claim 9 , wherein the at least one processing device is configured to generate the global tone map LUT based on the one or more parameters of the at least one imaging sensor.

15 . The electronic device of claim 9 , wherein, to generate the local tone mapped image, the at least one processing device is configured to:
apply the local tone map to the blended and demosaiced image to generate a first gain map; perform a morphological opening operation using the first gain map to generate a second gain map; perform a guided filtering operation using the first and second gain maps to generate a third gain map; and generate the local tone mapped image using the blended and demosaiced image and the third gain map.

16 . The electronic device of claim 9 , wherein:
the at least one processing device is configured to adjust the color saturation prior to performing a gamma correction; the at least one processing device is further configured to apply the gamma correction to the corrected image and convert from a red-green-blue (RGB) image domain to a luma-chroma (YUV) image domain, and to generate the output image, the at least one processing device is configured to apply contrast enhancement using the one or more contrast enhancement LUTs and apply global tone mapping using the global tone map LUT to a luma channel in the YUV image domain.

17 . A non-transitory machine re

Assignees

• Samsung Electronics Co Ltd

Inventors

• Gnanasambandam Abhiram
• Glotzbach John W
• Nadir Zeeshan
• Godaliyadda Gunawath Dilshan
• Lee John Seokjun
• Sheikh Hamid R

Classifications

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/20012

  Locally adaptive · CPC title

• G06T2207/10144

  Varying exposure · CPC title