Method and apparatus for inverse tone mapping

Patent metadata

Field	Value
Publication number	US-2022012855-A1
Application number	US-202117482433-A
Country	US
Kind code	A1
Filing date	Sep 23, 2021
Priority date	Dec 6, 2017
Publication date	Jan 13, 2022
Grant date	—

Abstract

In this invention, we propose a convolutional neural network (CNN) based architecture designed for the ITM to HDR consumer displays, called ITM-CNN, and its training strategy for enhancing the performance based on image decomposition using the guided filter. We demonstrate the benefits of decomposing the image by experimenting with various architectures and also compare the performance for different training strategies. To the best of our knowledge, this invention first presents the ITM problem using CNNs for HDR consumer displays, where the network is trained to restore lost details and local contrast. Our ITM-CNN can readily up-convert LDR images for direct viewing on an HDR consumer medium, and is a very powerful means to solve the lack of HDR video contents with legacy LDR videos.

First claim

1 . An image processing method, the method comprising:
receiving an image; decomposing the image into two or more decomposed images by performing a decomposition operation; extracting the feature maps for the decomposed images by performing convolution operations separately for the decomposed images; and generating a high dynamic range (HDR) image by processing the extracted feature maps.

2 . The method of claim 1 , wherein decomposing the image into two or more decomposed images comprises:
applying a guided filter for the image; and producing the two or more decomposed images as having different signal components.

3 . The method of claim 1 , wherein decomposing the image into two or more decomposed images comprises:
performing the convolutions for the image to yield the feature maps as input into two or more separate branches of convolution passes.

4 . The method of claim 1 , wherein generating an HDR image by processing the extracted feature maps comprises:
applying convolution operations on the extracted feature maps through one or more convolution layers.

5 . The method of claim 1 , wherein generating an HDR image by processing the extracted feature maps comprises:
multiplying the outputs of the different branches of the convolution passes.

6 . The method of claim 1 , further comprising:
training filter parameters of the convolution operations for at least one of image decomposition, feature extraction and HDR image generation.

7 . The method of claim 6 , wherein training the filter parameters comprises:
updating the filter parameter values for the convolution operations based on one or more training loss functions.

8 . The method of claim 7 , wherein updating the filter parameter values for the convolution operations based on one or more training loss functions comprises:
changing the filter parameter values for the convolutions based on the difference signals between predicted HDR image and the ground truth HDR image.

9 . The method of claim 2 , wherein producing the two or more decomposed images as having different signal components comprises generating the two or more decomposed images by performing edge-preserving filtering on the image.

10 . The method of claim 2 , wherein producing the two or more decomposed images as having different signal components comprises generating the two or more decomposed images by performing an element-wise division on the image by one of the two or more decomposed images.

11 . An image processing apparatus, the apparatus comprising:
a receiver configured to receive an image; and a processor configured to decompose the image into two or more decomposed images by performing a decomposition operation, to extract the feature maps by perfoiuiing convolution operations through one or more convolution layers for the decomposed images, and to generate a high dynamic range (HDR) image by processing the extracted feature maps.

12 . The apparatus of claim 11 , wherein decomposing the image into two or more decomposed images comprises:
applying a guided filter for the image; and producing the two or more decomposed images as having different signal components.

13 . The apparatus of claim 11 , wherein decomposing the image into two or more decomposed images comprises:
performing the convolutions for the image to yield the feature maps as input into two or more separate branches of convolution passes.

14 . The apparatus of claim 11 , wherein generating an HDR image by processing the extracted feature maps comprises:
applying convolution operations on the extracted feature maps through one or more convolution layers.

15 . The apparatus of claim 11 , wherein generating an HDR image by processing the extracted feature maps comprises:
multiplying the outputs of the different branches of the convolution passes.

16 . The apparatus of claim 11 , further comprising:
training filter parameters of the convolution operations for image decomposition, feature extraction and HDR image generation.

17 . The apparatus of claim 16 , wherein training the filter parameters comprises:
updating the filter parameter values for the convolution operations based on one or more training loss functions.

18 . The apparatus of claim 17 , wherein updating the filter parameter values for the convolution operations based on one or more training loss functions comprises:
changing the filter parameter values for the convolutions based on the difference signals between predicted HDR image and the ground truth HDR image.

19 . The apparatus of claim 12 , wherein producing the two or more decomposed images as having different signal components comprises generating the two or more decomposed images by performing edge-preserving filtering on the image.

20 . The apparatus of claim 12 , wherein producing the two or more decomposed images as having different signal components comprises generating the two or more decomposed images by performing an element-wise division on the image by one of the two or more decomposed images.

Assignees

• Korea Advanced Inst Sci & Tech

Inventors

• Kim Mun Churl
• Kim Soo Ye
• Kim Dae Eun

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10024

  Color image · CPC title