Super resolution using a generative adversarial network

Patent metadata

Field	Value
Publication number	US-2021264568-A1
Application number	US-202117302537-A
Country	US
Kind code	A1
Filing date	May 5, 2021
Priority date	Sep 15, 2016
Publication date	Aug 26, 2021
Grant date	—

Abstract

A neural network is trained to process received visual data to estimate a high-resolution version of the visual data using a training dataset and reference dataset. A set of training data is generated, and a generator convolutional neural network parameterized by first weights and biases is trained by comparing characteristics of the training data to characteristics of the reference dataset. The first network is trained to generate super-resolved image data from low-resolution image data and the training includes modifying first weights and biases to optimize processed visual data based on the comparison between the characteristics of the training data and the characteristics of the reference dataset. A discriminator convolutional neural network parameterized by second weights and biases is trained by comparing characteristics of the generated super-resolved image data to characteristics of the reference dataset, and where the second network is trained to discriminate super-resolved image data from real image data.

First claim

What is claimed is:

1 . A method comprising:
receiving an initial image; generating a super-resolution image from the initial image by using a generator convolutional neural network trained to minimize perceptual loss from the initial image, the generator convolutional neural network being parameterized by first weights and biases selected to optimize processed visual data based on the comparison between the one or more characteristics of visual image training data and the one or more characteristics of a visual image reference dataset and by using a discriminator convolutional neural network that is parameterized by second weights and biases, wherein the discriminator convolutional neural network is trained to discriminate super-resolved image data from real image data; and storing the generated super-resolution image.

2 . The method of claim 1 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a generator convolutional neural network that minimizes a Euclidean distance between feature representations of an image that is reconstructed from a downsampled version of a reference image and the reference image.

3 . The method of claim 1 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss and adversarial loss.

4 . The method of claim 1 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss, adversarial loss, and regularization loss.

5 . The method of claim 1 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a visual geometry group neural network.

6 . A computer-readable medium storing generator convolutional neural network and a discriminator convolutional neural network trained to generate an image using a method comprising:
receiving an initial image;
generating a super-resolution image from the initial image by using a generator convolutional neural network trained to minimize perceptual loss from the initial image, the generator convolutional neural network being parameterized by first weights and biases selected to optimize processed visual data based on the comparison between the one or more characteristics of visual image training data and the one or more characteristics of a visual image reference dataset and by using a discriminator convolutional neural network that is parameterized by second weights and biases, wherein the discriminator convolutional neural network is trained to discriminate super-resolved image data from real image data; and
storing the generated super-resolution image.

7 . The computer-readable medium of claim 6 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a generator convolutional neural network that minimizes a Euclidean distance between feature representations of an image that is reconstructed from a downsampled version of a reference image and the reference image.

8 . The computer-readable medium of claim 6 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss and adversarial loss.

9 . The computer-readable medium of claim 6 , wherein the generator convolutional neural network uses a perceptual loss function that is a weighted combination of content loss, adversarial loss, and regularization loss.

10 . The computer-readable medium of claim 6 , wherein using the generator convolutional neural network trained to minimize perceptual loss includes using a visual geometry group neural network.

Assignees

• Twitter Inc

Inventors

• Shi Wenzhe
• Ledig Christian
• Wang Zehan
• Theis Lucas
• Huszar Ferenc

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title