Image super-resolution reconstructing

Patent metadata

Field	Value
Publication number	US-12511713-B2
Application number	US-202117927863-A
Country	US
Kind code	B2
Filing date	May 10, 2021
Priority date	Jun 30, 2020
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

According to implementations of the subject matter described herein, a solution is proposed for super-resolution image reconstructing. According to the solution, an input image with first resolution is obtained. An invertible neural network is trained using the input image, wherein the invertible neural network is configured to generate an intermediate image with second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution. Subsequently, an output image with third resolution is generated based on the input image and second high-frequency information by using an inverse network of the trained invertible neural network, the second high-frequency information conforming to a predetermined distribution, and the third resolution being higher than the first resolution. The solution can effectively process a low-resolution image obtained by an unknown downsampling method, thereby obtaining a high-quality and high-resolution image.

First claim

The invention claimed is:

1 . A computer-implemented method, comprising:
obtaining an input image of a first resolution; training an invertible neural network with the input image, the invertible neural network being configured to generate an intermediate image of a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; wherein training an invertible neural network with the input image comprises:
determining a plurality of target functions based on the input image and the intermediate image by determining a first target function based on a difference between a pixel distribution in the intermediate image and a pixel distribution in an image block of the input image, the image block being of the second resolution; and
generating, using an inverse network of the trained invertible neural network, an output image of a third resolution based on second high-frequency information conforming to a predetermined distribution and the input image, the third resolution being greater than the first resolution.

2 . The method of claim 1 , wherein training an invertible neural network with the input image further comprises:
determining a total target function for training the invertible neural network by combining at least some of the plurality of target functions; and determining network parameters for the invertible neural network by minimizing the total target function.

3 . The method of claim 2 , wherein determining the first target function comprises:
distinguishing, by using a discriminator, whether a pixel of the intermediate image belongs to the intermediate image or the image block; and determining the first target function based on the distinguishing.

4 . The method of claim 2 , wherein determining a plurality of target functions comprises:
determining a second target function based on a difference between a distribution of the first high-frequency information and the predetermined distribution.

5 . The method of claim 2 , wherein determining a plurality of target functions comprises:
generating, by using an inverse network of the invertible neural network, a reconstructed image of the first resolution based on third high-frequency information conforming to the predetermined distribution and the intermediate image; and determining a third target function based on a difference between the input image and the reconstructed image.

6 . The method of claim 2 , wherein determining the plurality of target functions comprises:
obtaining a reference image corresponding to semantics of the input image, the reference image being of the second resolution; and determining a fourth target function based on a difference between the intermediate image and the reference image.

7 . The method of claim 1 , wherein the invertible neural network comprises a transforming module and at least one invertible network unit, and wherein generating the output image comprises:
generating, based on the input image and the second high-frequency information and by using the at least one invertible network unit, a low-frequency component and a high-frequency component to be merged, the low-frequency component representing semantics of the input image and the high-frequency component being related to the semantics; and merging, by using the transforming module, the low-frequency component and the high-frequency component into the output image.

8 . The method of claim 7 , wherein the transforming module comprises at least one of:
an invertible convolution block; and a wavelet transforming module.

9 . A device, comprising:
a processing unit; and a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising: obtaining an input image of a first resolution; training an invertible neural network with the input image, the invertible neural network being configured to generate an intermediate image of a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; wherein training an invertible neural network with the input image comprises:
determining a plurality of target functions based on the input image and the intermediate image by determining a first target function based on a difference between a pixel distribution in the intermediate image and a pixel distribution in an image block of the input image, the image block being of the second resolution; and
generating, using an inverse network of the trained invertible neural network, an output image of a third resolution based on second high-frequency information conforming to a predetermined distribution and the input image, the third resolution being greater than the first resolution.

10 . The device of claim 9 , wherein training an invertible neural network with the input image further comprises:
determining a total target function for training the invertible neural network by combining at least some of the plurality of target functions; and determining network parameters for the invertible neural network by minimizing the total target function.

11 . The device of claim 10 , wherein determining the first target function comprises:
distinguishing, by using a discriminator, whether a pixel of the intermediate image belongs to the intermediate image or the image block; and determining the first target function based on the distinguishing.

12 . The device of claim 10 , wherein determining a plurality of target functions comprises:
determining a second target function based on a difference between a distribution of the first high-frequency information and the predetermined distribution.

13 . A computer program product being tangibly stored in a non-transitory computer readable storage medium and comprising machine-executable instructions which, when executed by a device, cause the device to perform acts comprising:
obtaining an input image of a first resolution; training an invertible neural network with the input image, the invertible neural network being configured to generate an intermediate image of a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; wherein training an invertible neural network with the input image comprises:
determining a plurality of target functions based on the input image and the intermediate image by determining a first target function based on a difference between a pixel distribution in the intermediate image and a pixel distribution in an image block of the input image, the image block being of the second resolution; and
generating, using an inverse network of the trained invertible neural network, an output image of a third resolution based on second high-frequency information conforming to a predetermined distribution and the input image, the third resolution being greater than the first resolution.

14 . The computer program product of claim 13 , wherein training an invertible neural network with the input image further comprises:
determining a total target function for training the invertible neural network by combining at least some of the plurality of target functions; and determining network parameters for the invertible neural network by minimizing the total target function.

15 . The computer program product of claim 14 , wherein determining the first target function comprises:
distinguishing, by using a discriminator, whet

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Zheng Shuxin
• Liu Chang
• He Di
• Ke Guolin
• Bian Jiang
• Liu Tie-Yan

Classifications

• G06T3/4046

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title