Iterative multiscale image generation using neural networks

Patent metadata

Field	Value
Publication number	US-11734797-B2
Application number	US-202217751359-A
Country	US
Kind code	B2
Filing date	May 23, 2022
Priority date	Feb 24, 2017
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

A method of generating an output image having an output resolution of N pixels×N pixels, each pixel in the output image having a respective color value for each of a plurality of color channels, the method comprising: obtaining a low-resolution version of the output image; and upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations: obtaining a current version of the output image having a current K×K resolution; and processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having a 2K×2K resolution.

First claim

What is claimed is:

1. A method of generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the method comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution,

wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes columns of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes rows of pixels from the updated version of the output image.

2. The method of claim 1 , wherein obtaining the low-resolution version comprises:
generating the low-resolution version using an image generation machine learning model.

3. The method of claim 2 , wherein the image generation machine learning model is an autoregressive image generation machine learning model.

4. The method of claim 2 , wherein the output image is conditioned on an input context and wherein the image generation machine learning model is configured to generate the low-resolution version conditioned on the input context.

5. The method of claim 4 , wherein each convolutional neural network in each set of convolutional neural networks is conditioned on the input context.

6. The method of claim 1 , wherein processing the current version of the output image using the set of convolutional neural networks that are specific to the current resolution to generate the updated version of the output image comprises:
processing the current version of the output image using the first convolutional neural network to generate the first output image; generating the intermediate version by merging the current version and the first output image; processing the intermediate version using the second convolutional neural network to generate the second output image; and generating the updated version by merging the intermediate version and the second output image.

7. The method of claim 6 , wherein merging the current version and the first output image comprises:
generating an intermediate image that includes K columns of pixels from the current image and K columns of pixels from the first output image by alternating columns of pixels from the current version with columns of pixels from the first output image.

8. The method of claim 6 , wherein merging the intermediate version and the second output image comprises:
generating an updated image that includes a plurality of rows of pixels from the intermediate version and a plurality of rows of pixels from the second output image by alternating rows of pixels from the intermediate version with rows of pixels from the second output image.

9. The method of claim 6 ,
wherein the one or more channels are ordered according to a channel order,
wherein the first convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the first output image conditioned (i) on the current version and (ii) on values for pixels in the first output image for any channels before the channel in the channel order and (iii) not on values for pixels in the first output image for any channels that are after the channel in the channel order, and
wherein the second convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the second output image conditioned (i) on the intermediate version and (ii) on values for pixels in the second output image for any channels before the channel in the channel order and (iii) not on values for pixels in the second output image for any channels that are after the channel in the channel order.

10. The method of claim 9 , wherein processing the current version using the first convolutional neural network to generate the first output image comprises:
iteratively processing the current version and values from the first output image that have already been generated to generate the first output image, and
wherein processing the intermediate version using the second convolutional neural network to generate the second output image comprises:
iteratively processing the intermediate version and the values from the second output image that have already been generated to generate the second output image.

11. A method of generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the method comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution, wherein the set of convolutional neural networks that are specific to the current resolution comprises:

a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes rows of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes columns of pixels from the updated version of the output image.

12. The method of claim 11 , wherein processing the current version of the output image using the set of convolutional neural networks that are specific to the current resolution to generate the updated version comprises:
processing the current version using the first convolutional neural network to generate the first output image;
generating the intermediate version by merging the current version and the first output image;
processing the intermediate version using the second convolutional neural network to generate the second output image; and
generating the updated version by merging the intermediate version and the second output image.

13. The method of claim 12 , wherein merging the current version and the first output image comprises:
generating an intermediate image that includes rows of pixels from the current image and rows

Assignees

• Deepmind Tech Ltd

Inventors

• Kalchbrenner Nal Emmerich
• Belov Daniel
• Colmenarejo Sergio Gomez
• Van Den Oord Aaron Gerard Antonius
• Wang Ziyu
• Gomes de Freitas Joao Ferdinando
• Reed Scott Ellison

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N20/00

  Machine learning · CPC title