Image enhancement via iterative refinement based on machine learning models

Patent metadata

Field	Value
Publication number	US-11769228-B2
Application number	US-202117391150-A
Country	US
Kind code	B2
Filing date	Aug 2, 2021
Priority date	Aug 2, 2021
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

A method includes receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image. The method also includes training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process. The method additionally includes outputting the trained neural network.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image;
training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process; and
outputting the trained neural network.

2. The computer-implemented method of claim 1 , wherein the applying of the forward Gaussian diffusion process comprises determining, for an iterative step, a scalar hyperparameter indicative of a variance of the Gaussian noise at the iterative step.

3. The computer-implemented method of claim 1 , wherein the iterative denoising of the input image comprises predicting a noise vector based on a variance of the Gaussian noise added during the forward Gaussian process.

4. The computer-implemented method of claim 1 , wherein the neural network is a convolutional neural network comprising a U-net architecture based on a denoising diffusion probabilistic (DDPM) model.

5. The computer-implemented method of claim 1 , wherein the input image is of a first resolution and the enhanced version of the input image is of a second resolution, and wherein the second resolution is greater than the first resolution.

6. The computer-implemented method of claim 1 , wherein the iterative denoising of the input image further comprises:
upsampling the input image to an enhanced version by applying bicubic interpolation.

7. The computer-implemented method of claim 1 , wherein the neural network comprises a plurality of cascading models.

8. The computer-implemented method of claim 7 , wherein the plurality of cascading models are chained together.

9. The computer-implemented method of claim 1 , wherein the iterative denoising of the input image further comprises:
a plurality of iterative refinement steps corresponding to different levels of image quality, and wherein each step is trained with a regression loss.

10. The computer-implemented method of claim 9 , wherein a number of the plurality of iterative refinement steps is greater for refinement steps corresponding to a lower image quality than for refinement steps corresponding to a higher image quality.

11. The computer-implemented method of claim 1 , wherein the neural network comprises a plurality of cascading models, and wherein the training of the neural network comprises training the plurality of cascading models in parallel.

12. The computer-implemented method of claim 1 , wherein the training of the neural network is performed at the computing device.

13. A computer-implemented method, comprising:
receiving, by a computing device, an input image;
applying a neural network to predict an enhanced version of the input image by iteratively denoising the input image, wherein the iterative denoising is based on a reverse Markov chain associated with a forward Gaussian diffusion process, the neural network having been trained by applying the forward Gaussian diffusion process to add Gaussian noise to at least one corresponding target version of each image of a plurality of pairs of images in training data; and
outputting the predicted enhanced version of the input image.

14. The computer-implemented method of claim 13 , wherein the neural network is a convolutional neural network comprising a U-net architecture based on a denoising diffusion probabilistic (DDPM) model.

15. The computer-implemented method of claim 13 , wherein the iterative denoising of the input image further comprising:
upsampling the input image to an enhanced version by applying bicubic interpolation.

16. The computer-implemented method of claim 13 , wherein the neural network comprises a plurality of cascading models.

17. The computer-implemented method of claim 16 , wherein the plurality of cascading models are chained together.

18. The computer-implemented method of claim 13 , wherein the outputting of the predicted enhanced version of the input image further comprising:
obtaining a trained neural network at the computing device; and
applying the trained neural network as obtained to the outputting of the predicted enhanced version of the input image.

19. The computer-implemented method of claim 13 , wherein the outputting of the predicted enhanced version of the input image further comprising:
determining, by the computing device, a request to predict the enhanced version of the input image;
sending the request from the computing device to a second computing device, the second computing device comprising a trained version of the neural network; and
after sending the request, the computing device receiving, from the second computing device, the predicted enhanced version of the input image.

20. The computer-implemented method of claim 13 , wherein the input image is of a first image resolution and the enhanced version of the input image is of a second image resolution, and wherein the second image resolution is greater than the first image resolution.

21. The computer-implemented method of claim 13 , wherein the input image is of a first color version and the enhanced version of the input image is of a second color version, and wherein the second color version is of a higher quality than the first color version.

22. The computer-implemented method of claim 13 , wherein the input image is of a first light composition and the enhanced version of the input image is of a second light composition, and wherein the second light composition is of a higher quality than the first light composition.

23. The computer-implemented method of claim 13 , wherein the input image comprises video content.

24. The computer-implemented method of claim 13 , wherein the input image is a compressed image file, and wherein the predicted enhanced version is a decompressed version of the compressed image file.

25. A computing device, comprising:
one or more processors; and
data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out operations comprising:
receiving, by the computing device, an input image;
applying a neural network to predict an enhanced version of the input image by iteratively denoising the input image, wherein the iterative denoising is based on a reverse Markov chain associated with a forward Gaussian diffusion process, the neural network having been trained by applying the forward Gaussian diffusion process to add Gaussian noise to at least one corresponding target version of each image of a plurality of pairs of images in training data; and
outputting the predicted enhanced version of the input image.

Assignees

• Google Llc

Inventors

• Saharia Chitwan
• Ho Jonathan
• Chan William
• Salimans Tim
• Fleet David
• Norouzi Mohammad

Classifications

• G06T5/002 Primary

  Physics · mapped topic

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title