Low-light video processing method, device and storage medium

Patent metadata

Field	Value
Publication number	US-2023196721-A1
Application number	US-202117999467-A
Country	US
Kind code	A1
Filing date	Jun 9, 2021
Priority date	Jun 18, 2020
Publication date	Jun 22, 2023
Grant date	—

Abstract

A low-illuminance video processing method, a low-illuminance video processing device and a storage medium are disclosed. The method includes: acquiring a same number of preceding frame images and subsequent frame images corresponding to a current video frame of a low-illuminance video to obtain a frame image set corresponding to the current video frame, and performing traversal on the low-illuminance video to obtain frame image sets corresponding to all video frames; after performing image alignment on all frame images in the frame image sets corresponding to all video frames, inputting the frame image sets into a pre-trained low-illuminance image enhancement model to obtain enhanced frame images; and generating an enhanced video based on the enhanced frame images.

First claim

1 . A low-illuminance video processing method, comprising:
acquiring a same number of preceding frame images and subsequent frame images corresponding to a current video frame of a low-illuminance video to obtain a frame image set corresponding to the current video frame, and performing traversal on the low-illuminance video to obtain frame image sets corresponding to all video frames; after performing image alignment on all frame images in the frame image sets corresponding to all video frames, inputting the frame image sets into a pre-trained low-illuminance image enhancement model to obtain enhanced frame images; and generating an enhanced video based on the enhanced frame images.

2 . The method of claim 1 , wherein performing image alignment on all frame images in the frame image sets corresponding to all video frames comprises:
using a target video frame among all video frames as a reference frame image to perform motion estimation on each supplementary frame image in the frame image set corresponding to the target video frame, so as to sequentially determine a motion vector corresponding to each supplementary frame image in the frame image set corresponding to the target video frame, the supplementary frame images being the preceding frame images and the subsequent frame images in the frame image set and the target video frame being each video frame among all video frames; respectively performing motion compensation on each of the supplementary frame images based on the motion vector corresponding to each of the supplementary frame images, so as to determine each supplementary frame image subjected to motion compensation; and concatenating the target video frame and each of the supplementary frame images subjected to motion compensation according to channels to obtain an aligned frame image set corresponding to the target video frame.

3 . The method of claim 2 , wherein using a target video frame among all video frames as a reference frame image to perform motion estimation on each supplementary frame image in the frame image set corresponding to the target video frame, so as to sequentially determine a motion vector corresponding to each supplementary frame image in the frame image set corresponding to the target video frame comprises:
for each supplementary frame image in the frame image set corresponding to the target video frame, inputting the target video frame and the supplementary frame image into a pre-trained motion estimation model to obtain a motion vector corresponding to the supplementary frame image.

4 . The method of claim 2 , wherein using a target video frame among all video frames as a reference frame image to perform motion estimation on each supplementary frame image in the frame image set corresponding to the target video frame, so as to sequentially determine a motion vector corresponding to each supplementary frame image in the frame image set corresponding to the target video frame comprises:
inputting the target video frame and the preceding frame image into a pre-trained preceding frame image motion estimation model to obtain a motion vector corresponding to the preceding frame image; and inputting the target video frame and the subsequent frame image into a pre-trained subsequent frame image motion estimation model to obtain a motion vector corresponding to the subsequent frame image.

5 . The method of claim 2 , wherein the respectively performing motion compensation on each of the supplementary frame images based on the motion vector corresponding to each of the supplementary frame images, so as to determine each supplementary frame image subjected to motion compensation comprises:
based on the motion vector corresponding to each supplementary frame image, respectively performing warping on each of the supplementary frame images, so as to determine each supplementary frame image subjected to motion compensation.

6 . The method of claim 1 , wherein the low-illuminance image enhancement model comprises:
a decomposition sub-model, a reflectance map enhancement sub-model, and an illumination map enhancement sub-model; and after performing image alignment on all frame images in the frame image sets corresponding to all video frames, inputting the frame image sets into a pre-trained low-illuminance image enhancement model to obtain enhanced frame images comprises:
after performing image alignment on all frame images in a target frame image set corresponding to the target video frame among all video frames, inputting the target frame image set into the decomposition sub-model to obtain a reflectance component map and an illumination component map corresponding to the target frame image set, the target video frame being each video frame among all video frames;
inputting the reflectance component map into the reflectance map enhancement sub-model to obtain an enhanced reflectance component map;
inputting the illumination component map into the illumination map enhancement sub-model to obtain an enhanced illumination component map; and
multiplying the enhanced reflectance component map and the enhanced illumination component map to obtain enhanced frame images corresponding to the target video frame.

7 . The method of claim 6 , wherein before inputting the target frame image set into the decomposition sub-model to obtain a reflectance component map and an illumination component map corresponding to the target frame image set, the method further comprises:
adjusting a resolution of each of the frame images subjected to image alignment in the target frame image set from an original resolution to a resolution threshold to obtain a resolution-adjusted target frame image set, the resolution threshold being less than the original resolution;
wherein inputting the target frame image set into the decomposition sub-model to obtain a reflectance component map and an illumination component map corresponding to the target frame image set comprises:
inputting the resolution-adjusted target frame image set into the decomposition sub-model to obtain a reflectance component map and an illumination component map corresponding to the target frame image set;

wherein after inputting the target frame image set into the decomposition sub-model to obtain a reflectance component map and an illumination component map corresponding to the target frame image set, the method further comprises:
adjusting a resolution of the reflectance component map to the original resolution to obtain a reflectance component map with the original resolution; and
adjusting a resolution of the illumination component map to the original resolution to obtain an illumination component map with the original resolution;

wherein inputting the reflectance component map into the reflectance map enhancement sub-model to obtain an enhanced reflectance component map comprises:
inputting the reflectance component map with the original resolution into the reflectance map enhancement sub-model to obtain an enhanced reflectance component map; and

wherein inputting the illumination component map into the illumination map enhancement sub-model to obtain an enhanced illumination component map comprises:
inputting the illumination component map with the original resolution into the illumination map enhancement sub-model to obtain an enhanced illumination component map.

8 . The method of claim 1 , wherein acquiring a same number of preceding frame images and subsequent frame images corresponding to a current video frame of a low-illuminance video comprises:
acquiring one preceding frame image and one subsequent frame image corresponding to a current video frame of a low-illuminance video.

9 . The method of claim 8 , wherein in response to that

Assignees

• Sanechips Tech Co Ltd

Inventors

• Ren Cong
• Xu Ke
• Kong Dehui
• You Jing
• Liu Xin
• Zhu Fang

Classifications

• G06V10/60 Primary

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06V10/24 Primary

  Aligning, centring, orientation detection or correction of the image · CPC title

• G06T3/0093

  Physics · mapped topic

• H04N19/513

  Processing of motion vectors · CPC title