Techniques for domain to domain projection using a generative model

Patent metadata

Field	Value
Publication number	US-11880766-B2
Application number	US-202117384357-A
Country	US
Kind code	B2
Filing date	Jul 23, 2021
Priority date	Oct 16, 2020
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

An improved system architecture uses a pipeline including a Generative Adversarial Network (GAN) including a generator neural network and a discriminator neural network to generate an image. An input image in a first domain and information about a target domain are obtained. The domains correspond to image styles. An initial latent space representation of the input image is produced by encoding the input image. An initial output image is generated by processing the initial latent space representation with the generator neural network. Using the discriminator neural network, a score is computed indicating whether the initial output image is in the target domain. A loss is computed based on the computed score. The loss is minimized to compute an updated latent space representation. The updated latent space representation is processed with the generator neural network to generate an output image in the target domain.

First claim

The invention claimed is:

1. A computer-implemented method for generating an image using a generative adversarial network comprising a generator neural network and a discriminator neural network, the method comprising:
obtaining an input image in a first domain and information about a target domain, wherein the first and target domains correspond to image styles;
producing an initial latent space representation of the input image by encoding the input image;
generating an initial output image by processing the initial latent space representation of the input image with the generator neural network;
computing, using the discriminator neural network, a score indicating whether the initial output image is in the target domain;
computing a loss based on the computed score;
minimizing the loss to compute an updated latent space representation of the input image; and
processing the updated latent space representation of the input image with the generator neural network to generate an output image in the target domain.

2. The method of claim 1 , wherein the loss further comprises a difference between the initial latent space representation and a target latent code.

3. The method of claim 2 , wherein the target latent code comprises a mean latent code from a training phase of the generator neural network.

4. The method of claim 1 , wherein the encoding is performed using an encoder neural network, the method further comprising training the encoder neural network on randomly-generated synthetic images mapped from a Gaussian distribution.

5. The method of claim 4 , wherein the Gaussian distribution is truncated at a value between 0.6 and 0.8.

6. The method of claim 1 , further comprising:
displaying a user interface; and
receiving input to the user interface to generate a collage using a set of initial images,
wherein the generated collage is the input image, and
wherein the output image is a photorealistic image generated from the collage.

7. The method of claim 1 , wherein the loss further comprises a pixel loss component and a perceptual loss component.

8. A computing system comprising:
a processor;
a non-transitory computer-readable medium comprising instructions, including a generative adversarial network comprising a generator neural network and a discriminator neural network, the instructions which, when executed by the processor, perform processing comprising:
obtaining an input image in a first domain and information about a target domain, wherein the first and target domains correspond to image styles;
producing an initial latent space representation of the input image by encoding the input image;
generating an initial output image by processing the initial latent space representation of the input image with the generator neural network;
computing, using the discriminator neural network, a score indicating whether the initial output image is in the target domain;
computing a loss based on the computed score;
minimizing the loss to compute an updated latent space representation of the input image; and
processing the updated latent space representation of the input image with the generator neural network to generate an output image in the target domain.

9. The computing system of claim 8 , wherein the loss further comprises a difference between the initial latent space representation and a target latent code.

10. The computing system of claim 9 , wherein the target latent code comprises a mean latent code from a training phase of the generator neural network.

11. The computing system of claim 8 , wherein the encoding is performed using an encoder neural network, the processing further comprising training the encoder neural network on randomly-generated synthetic images mapped from a Gaussian distribution.

12. The computing system of claim 11 , wherein the Gaussian distribution is truncated at a value between 0.6 and 0.8.

13. The computing system of claim 8 , the processing further comprising:
displaying a user interface; and
receiving input to the user interface to generate a collage using a set of initial images,
wherein the generated collage is the input image, and
wherein the output image is a realistic image generated from the collage.

14. The computing system of claim 8 , wherein the loss further comprises a pixel loss component and a perceptual loss component.

15. A non-transitory computer-readable medium having instructions stored thereon, the instructions executable by a processing device to perform operations for generating an image using a generative adversarial network comprising a generator neural network and a discriminator neural network, the operations comprising:
obtaining an input image in a first domain and information about a target domain, wherein the first and target domains correspond to image styles;
producing an initial latent space representation of the input image by encoding the input image;
a step for updating the initial latent space representation by minimizing a loss based on score generated using the discriminator neural network; and
processing the updated latent space representation with the generator neural network to generate an output image in the target domain.

16. The medium of claim 15 , wherein the loss further comprises a difference between the initial latent space representation and a target latent code.

17. The medium of claim 16 , wherein the target latent code comprises a mean latent code from a training phase of the generator neural network.

18. The medium of claim 15 , wherein the encoding is performed using an encoder neural network, the operations further comprising training the encoder neural network on randomly-generated synthetic images mapped from a Gaussian distribution.

19. The medium of claim 18 , wherein the Gaussian distribution is truncated at a value between 0.6 and 0.8.

20. The medium of claim 15 , the operations further comprising:
displaying a user interface; and
receiving input to the user interface to generate a collage using a set of initial images,
wherein the generated collage is the input image, and
wherein the output image is a photorealistic image generated from the collage.

Assignees

• Adobe Inc

Inventors

• Smith Cameron
• Kalarot Ratheesh
• Lin Wei-An
• Zhang Richard
• Mitra Niloy
• Shechtman Elya
• Ghadar Shabnam
• Shu Zhixin
• Hold-Geoffrey Yannick
• Carr Nathan
• Lu Jingwan
• Wang Oliver
• Zhu Jun-Yan

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/094 Primary

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title