Method and device for mapping LDR video into HDR video

Patent metadata

Field	Value
Publication number	US-11436710-B2
Application number	US-202017076242-A
Country	US
Kind code	B2
Filing date	Oct 21, 2020
Priority date	Jun 18, 2020
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

A method for mapping a low-dynamic range (LDR) video into a high-dynamic range (HDR) video and a device therefor are provided. The method includes a modeling process, and a mapping process. The modeling process includes training according to LDR videos of at least three different exposure levels to obtain a highlight reconstruction model and an exposure generation model, and the mapping process includes mapping an LDR video to be processed into an HDR video through the highlight reconstruction model and the exposure generation model. Accordingly, solutions to the problems of a multi-exposure synthetic image being too dark or too light is provided, and jitter in video synthesis, are provided.

First claim

What is claimed is:

1. A method for mapping a low-dynamic range (LDR) video into a high-dynamic range (HDR) video, the method comprising:
a modeling process; and
a mapping process,
wherein the modeling process comprises training according to LDR videos of at least three different exposure levels to obtain a highlight reconstruction model and an exposure generation model, and
wherein the mapping process comprises:
decoding an LDR video to be processed and inputting the decoded LDR video to the highlight reconstruction model to obtain highlighted reconstructed images of respective LDR video frames in the LDR video,
inputting the highlighted reconstructed images to the exposure generation model to obtain at least three frames of different exposure levels corresponding to each reconstructed image,
synthesizing the at least three frames of the different exposure levels corresponding to each reconstructed image into a frame of HDR image, and
encoding all HDR images according to a time sequence and synthesizing the HDR images into an HDR video.

2. The method according to claim 1 , wherein the modeling process further comprises:
shooting a same scene with the at least three different exposure levels simultaneously to obtain the LDR videos of the at least three different exposure levels,
performing scene detection on the LDR videos of the at least three different exposure levels respectively, and marking frames in which a scene change occurs in the LDR videos,
extracting frames between a start frame and an end frame of each scene according to densities from high to low,
searching, on each of the extracted frames, for a highlight region in the frame, and training a highlight reconstruction model with a two-dimensional convolutional neural network based on the highlight region and frames before and after the frame, to obtain a highlighted reconstructed image of the frame, and
training an exposure generation model with a three-dimensional convolutional neural network based on the obtained highlighted reconstructed image according to an order of the obtained highlighted reconstructed image in an original LDR video, to obtain at least three frames of different exposure levels corresponding to each reconstructed image, respectively.

3. The method according to claim 2 , wherein the at least three different exposure levels comprise:
⅓ of camera aperture and shutter time limits, as a low exposure level,
⅔ of camera aperture and shutter time limits, as a medium exposure level, and
3/3 of camera aperture and shutter time limits, as a high exposure level.

4. The method according to claim 2 , wherein the performing of the scene detection comprises:
searching for a matching block by a motion search, and
comparing a mean square error of the matching block with a set threshold to determine whether a scene change occurs in a current frame.

5. The method according to claim 2 , wherein the extracting of the frames between the start frame and the end frame comprises at least one of:
gradually reducing the densities of the extracted frames according to a geometric progression from a start frame of each scene until the end frame of the scene, or
gradually reducing the densities of the extracted frames according to a set non-geometric progression from the start frame of each scene until the end frame of the scene.

6. The method according to claim 2 ,
wherein the frames before and after the frame comprise:
frames in the extracted frames that are in the same scene as the frame, and within T1 frames before the frame and T2 frames after the frame, and wherein T1 and T2 are preset positive integers.

7. A device for mapping an LDR video into an HDR video, the device comprising:
a memory; and
a at least one processor,
wherein the at least one processor is configured to perform training according to LDR videos of at least three different exposure levels to obtain a highlight reconstruction model and an exposure generation model, and
wherein the at least one processor is configured to:
decode an LDR video to be processed and input the decoded LDR video to the highlight reconstruction model to obtain highlighted reconstructed images of respective LDR video frames in the LDR video,
input the highlighted reconstructed images to the exposure generation model to obtain at least three frames of different exposure levels corresponding to each reconstructed image,
synthesize the at least three frames of the different exposure levels corresponding to each reconstructed image into a frame of HDR image, and
encode all HDR images according to a time sequence and synthesize the HDR images into an HDR video.

8. The device according to claim 7 , wherein the at least one processor is further configured to:
shoot a same scene with the at least three different exposure levels simultaneously to obtain the LDR videos of the at least three different exposure levels,
perform scene detection on the LDR videos of the at least three different exposure levels respectively, and mark a frame in which a scene change occurs in the LDR videos,
extract frames between a start frame and an end frame of each scene according to densities from high to low,
search, on each of the extracted frames, for a highlight region in the frame, and train a highlight reconstruction model with a two-dimensional convolutional neural network based on the highlight region and frames before and after the frame, to obtain a highlighted reconstructed image of the frame, and
train an exposure generation model with a three-dimensional convolutional neural network based on the obtained highlighted reconstructed image according to an order of the obtained highlighted reconstructed image in an original LDR video, to obtain at least three frames of different exposure levels corresponding to each reconstructed image, respectively.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Wang Zuolong

Classifications

• H04N23/73

  by influencing the exposure time · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title