Super resolution using a generative adversarial network

Patent metadata

Field	Value
Publication number	US-11024009-B2
Application number	US-201715706428-A
Country	US
Kind code	B2
Filing date	Sep 15, 2017
Priority date	Sep 15, 2016
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

A neural network is trained to process received visual data to estimate a high-resolution version of the visual data using a training dataset and reference dataset. A set of training data is generated and a generator convolutional neural network parameterized by first weights and biases is trained by comparing characteristics of the training data to characteristics of the reference dataset. The first network is trained to generate super-resolved image data from low-resolution image data and the training includes modifying first weights and biases to optimize processed visual data based on the comparison between the characteristics of the training data and the characteristics of the reference dataset. A discriminator convolutional neural network parameterized by second weights and biases is trained by comparing characteristics of the generated super-resolved image data to characteristics of the reference dataset, and where the second network is trained to discriminate super-resolved image data from real image data.

First claim

What is claimed is:

1. A method for training a neural network to process at least a section of received low-resolution visual data to estimate a high-resolution version of the low-resolution visual data using a training dataset and reference dataset, the method comprising:
(a) generating a set of training data;
(b) training a generator convolutional neural network that is parameterized by first weights and biases by comparing one or more characteristics of the training data to one or more characteristics of at least a section of the reference dataset, wherein the generator convolutional neural network is trained to generate super-resolved image data from low-resolution image data and wherein the training includes modifying one or more of the first weights and biases of the generator convolutional neural network to optimize processed visual data based on the comparison between the one or more characteristics of the training data and the one or more characteristics of the reference dataset, wherein the modification is based on a perceptual loss function that includes a weighted combination of a content loss function, an adversarial loss function based on a discriminator network trained to differentiate between the super-resolved images and original photo-realistic images, and a regularization loss function that encourages spatially coherent solutions, wherein a first feature map of the neural network is generated by the neural network from the low-resolution image data and a second feature map of the neural network is generated from the super-resolved reference image data and wherein the content loss function is based on a Euclidean distance between the first and second feature maps, where similarities of the feature maps of the neural network are based on human notions of content similarity as determined based on object classification of training; and
(c) training a discriminator convolutional neural network that is parameterized by second weights and biases by comparing one or more characteristics of the generated super-resolved image data to one or more characteristics of at least a section of the reference dataset, wherein the second network is trained to discriminate super-resolved image data from real image data.

2. The method of claim 1 , wherein the training dataset includes a plurality of visual images.

3. The method of claim 1 , wherein the reference dataset includes a plurality of visual images.

4. The method of claim 3 , wherein the plurality of visual images of the reference dataset are not increased quality versions of the visual data of the training dataset.

5. The method of claim 1 , further comprising:
generating an estimated high-resolution version of an input image by using the trained convolutional neural network on the input image, wherein the trained convolutional neural network is configured to remove compression artifacts from the input image to generate the estimated high-resolution version of an input image.

6. The method of claim 1 , further comprising:
generating an estimated high-resolution version of an input image by using the trained convolutional neural network on the input image, wherein the trained convolutional neural network is configured to perform image de-mosaicing on the input image to generate the estimated high-resolution version of an input image.

7. The method of claim 1 , further comprising:
generating an estimated high-resolution version of an input image by using the trained convolutional neural network on the input image, wherein the trained convolutional neural network is configured to perform image de-noising on the input image to generate the estimated high-resolution version of an input image.

8. The method of claim 1 , wherein the generator convolutional neural network is hierarchical and includes a plurality of layers.

9. The method of claim 8 , wherein the layers are any of sequential, recurrent, recursive, branching, or merging.

10. The method of claim 1 , further comprising:
iterating over (a), (b), and (c); and
updating the training data during an iteration.

Assignees

• Twitter Inc

Inventors

• Shi Wenzhe
• Ledig Christian
• Wang Zehan
• Theis Lucas
• Huszar Ferenc

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title