Image restoration method and apparatus, and electronic device

Patent metadata

Field	Value
Publication number	US-12469112-B2
Application number	US-202117922150-A
Country	US
Kind code	B2
Filing date	May 25, 2021
Priority date	Jun 22, 2020
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

An image restoration method and apparatus, and an electronic device. The method includes: inputting, into a target denoising network, an image to be processed, where the target denoising network includes a single-frame network and a recursive network, and the image to be processed is any frame in a video to be processed (S 101 ); removing, via the single-frame network, compression noise of the image to be processed, and outputting a first image (S 102 ); according to the content of the previous frame of image, removing, via the recursive network, compression noise of the image to be processed, and outputting a second image, where the previous frame of image is the previous frame of image of the image to be processed in the video to be processed (S 103 ); and performing weighted summation on the first image and the second image, and outputting a denoised image for the image to be processed (S 104 ).

First claim

What is claimed is:

1 . An image restoration method, comprising:
inputting an image to be processed into a target denoising network, wherein the target denoising network comprises a single-frame network and a recursive network, and the image to be processed is any frame in a video to be processed; removing, via the single-frame network, compression noise of the image to be processed to output a first image; removing, according to a content of a previous frame image, compression noise of the image to be processed via the recursive network to output a second image, wherein the previous frame image is one previous frame of the image to be processed in the video to be processed; and performing weighted summation on the first image and the second image, and outputting a denoised image for the image to be processed; wherein the removing, according to the content of the previous frame image, the compression noise of the image to be processed via the recursive network to output the second image, comprises: removing the compression noise of the image to be processed via at least one first convolution layer, at least one first feature series layer and at least one first sampling layer cascaded in the recursive network, to output the second image; wherein the at least one first convolution layer in the recursive network comprises a first sub-convolution layer and a second sub-convolution layer, the at least one first feature series layer comprises a first sub-feature series layer and a second sub-feature series layer, and the at least one first sampling layer comprises first down-sampling layers and first up-sampling layers; wherein the removing the compression noise of the image to be processed via the at least one first convolution layer, the at least one first feature series layer and the at least one first sampling layer cascaded in the recursive network, to output the second image comprises: receiving, via the first sub-feature series layer, a first feature image of the image to be processed extracted by each of third sub-convolution layers in second convolution layers in the single-frame network; obtaining, via the first sub-feature series layer, a second feature image extracted from the previous frame image by the first sub-convolution layer corresponding to each of the third sub-convolution layers in the recursive network; obtaining a series feature image by performing, via the first sub-feature series layer, series operation on the first feature image and the second feature image; obtaining a compressed feature image by performing, via each of first sub-convolution layers, compression on the series feature image, wherein the compressed feature images are second feature images extracted from the image to be processed by the first sub-convolution layers; extracting, via the first down-sampling layers in the at least one first sampling layer, feature images with a plurality of spatial sizes from the compressed feature images; determining, via the first up-sampling layers, feature images with same spatial sizes as the plurality of spatial sizes; obtaining a first splicing feature image by splicing, via the second sub-feature series layer, the feature images with the same spatial sizes on feature dimension; and processing, via the second sub-convolution layer, the first splicing feature image, and outputting the second image.

2 . The method according to claim 1 , wherein the single-frame network comprises at least one second convolution layer, at least one second sampling layer and at least one second feature series layer which are cascaded, the at least one second convolution layer comprises a third sub-convolution layer and a fourth sub-convolution layer, and the at least one second sampling layer comprises second down-sampling layers and second up-sampling layers;
wherein the removing, via the single-frame network, the compression noise of the image to be processed to output the first image, comprises: extracting, via third sub-convolution layers, first feature images of the image to be processed; extracting, via the second down-sampling layers, feature images with a plurality of spatial sizes from the first feature images; determining, via the second up-sampling layers, feature images with the same spatial sizes as the plurality of spatial sizes; obtaining a second splicing feature image by splicing, via the second feature series layer, the feature images with the same spatial sizes on feature dimension; and processing, via the fourth sub-convolution layer, the second splicing feature image, and outputting the first image.

3 . The method according to claim 2 , wherein before inputting the image to be processed into the target denoising network, the method further comprises:
a training process of the target denoising network, wherein the training process comprises: obtaining a plurality of groups of image frame sequences, wherein each group of image frame sequences comprises a plurality of images; encoding the plurality of groups of image frame sequences into a true value video and a simulation video respectively, wherein each frame of simulation image in the simulation video comprises compression noise; inputting the each frame of simulation image in the simulation video into a denoising network to be trained; outputting a simulation denoised image of a corresponding frame; determining, according to a first prediction deviation between the simulation denoised image and a true value image of the corresponding frame in the true value video, a first loss function for the denoising network to be trained; and taking a corresponding network when the first loss function is lower than a first preset threshold as the target denoising network.

4 . The method according to claim 1 , wherein before inputting the image to be processed into the target denoising network, the method further comprises:
a training process of the target denoising network, wherein the training process comprises: obtaining a plurality of groups of image frame sequences, wherein each group of image frame sequences comprises a plurality of images; encoding the plurality of groups of image frame sequences into a true value video and a simulation video respectively, wherein each frame of simulation image in the simulation video comprises compression noise; inputting the each frame of simulation image in the simulation video into a denoising network to be trained; outputting a simulation denoised image of a corresponding frame; determining, according to a first prediction deviation between the simulation denoised image and a true value image of the corresponding frame in the true value video, a first loss function for the denoising network to be trained; and taking a corresponding network when the first loss function is lower than a first preset threshold as the target denoising network.

5 . The method according to claim 4 , wherein the determining, according to the first prediction deviation between the simulation denoised image and the true value image of the corresponding frame in the true value video, the first loss function for the denoising network to be trained, comprises:
adopting an L2 loss function when the first prediction deviation between the simulation denoised image and the true value image of the corresponding frame in the true value video is smaller than or equal to δ; and adopting an L1 loss function when the first prediction deviation between the simulation denoised image and the true value image of the corresponding frame in the true value video is greater than δ; wherein a formula corresponding to the L2 loss function is:

L
δ

Assignees

• Boe Technology Group Co Ltd
• Beijing Boe Technology Dev Co Ltd

Inventors

• Duan Ran
• Zhu Dan

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T3/4046

  using neural networks · CPC title

• H04N19/86 Primary

  involving reduction of coding artifacts, e.g. of blockiness · CPC title