Supervised learning techniques for encoder training

Patent metadata

Field	Value
Publication number	US-12412089-B2
Application number	US-202117384378-A
Country	US
Kind code	B2
Filing date	Jul 23, 2021
Priority date	Oct 16, 2020
Publication date	Sep 9, 2025
Grant date	Sep 9, 2025

Abstract

Systems and methods train an encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The encoder is trained by providing an input training image to the encoder and producing, by the encoder, a latent space representation of the input training image. The latent space representation is provided as input to the GAN to generate a generated training image. A latent code is sampled from a latent space associated with the GAN and the sampled latent code is provided as input to the GAN. The GAN generates a synthetic training image based on the sampled latent code. The sampled latent code is provided as input to the encoder to produce a synthetic training code. The encoder is updated by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code.

First claim

The invention claimed is:

1. A computer-implemented method for training an encoder neural network comprising:
providing an input training image as input to the encoder neural network, the encoder neural network implemented by one or more computing systems;
producing, by the encoder neural network, a latent space representation of the input training image;
providing the latent space representation of the input training image produced by the encoder neural network as input to a generator neural network, the generator neural network implemented using the one or more computing systems;
generating, by the generator neural network, a generated training image based upon the latent space representation of the input training image;
sampling a latent code from a latent space associated with the generator neural network, wherein sampling the latent code from the latent space comprises:
sampling from a first latent space to obtain an initial latent code, and
transforming the initial latent code to an augmented latent space to generate the sampled latent code;
providing the sampled latent code as input to the generator neural network;

generating, by the generator neural network, a synthetic training image based on the sampled latent code;
providing the sampled latent code as input to the encoder neural network;
producing, by the encoder neural network, a synthetic training code based upon the sampled latent code;
updating the encoder neural network by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code to produce a trained encoder neural network; and
providing the trained encoder neural network and the generator neural network for generating an output image for display.

2. The method of claim 1 , wherein the loss between the generated training image and the input training image is weighted by a first value, and the loss between the synthetic training code and the sampled latent code is weighted by a second value different from the first value.

3. The method of claim 1 , further comprising sampling in a neighborhood of the transformed initial latent code to generate the sampled latent code.

4. The method of claim 1 , further comprising:
providing a second input training image as input to the encoder neural network to produce a latent space representation of the second input training image;
sampling a second latent code from the latent space by sampling in a neighborhood of the latent space representation of the second input training image; and
retraining the encoder neural network using the second sampled latent code.

5. The method of claim 4 , further comprising:
providing a third input training image as input to the encoder neural network to produce a latent space representation of the third input training image;
generating a third generated training image by providing the latent space representation of the third input training image as input to the generator neural network;
minimizing a loss between the latent space representation of the third input training image and the third generated training image to optimize the latent space representation of the third input training image;
sampling a third latent code from the latent space by sampling in a neighborhood of the optimized latent space representation of the third input training image; and

retraining the encoder neural network using the third sampled latent code.

6. The method of claim 5 , wherein retraining the encoder neural network using the third sampled latent code comprises:
providing the third sampled latent code as input to the generator neural network to generate a third synthetic training image;
providing the third synthetic training image as input to the encoder neural network to produce a third synthetic training code; and
updating the encoder neural network by minimizing a loss between the third generated training image and the third input training image, and the third synthetic training code and the third sampled latent code.

7. A computing system comprising:
a processor;
a non-transitory computer-readable medium comprising instructions which, when executed by the processor, perform processing comprising:
providing an input training image as input to an encoder neural network, the encoder neural network implemented by one or more computing systems;

producing, by the encoder neural network, a latent space representation of the input training image;
providing the latent space representation of the input training image produced by the encoder neural network as input to a generator neural network, the generator neural network implemented using the one or more computing systems;
generating, by the generator neural network, a generated training image based upon the latent space representation of the input training image;
generating a sampled latent code by:
sampling from a first latent space to obtain an initial latent code, and
transforming the initial latent code to an augmented latent space to generate the sampled latent code;

providing the sampled latent code as input to the generator neural network;
generating, by the generator neural network, a synthetic training image based on the sampled latent code;
providing the sampled latent code as input to the encoder neural network;
producing, by the encoder neural network, a synthetic training code based upon the sampled latent code; and
updating the encoder neural network by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code to produce a trained encoder neural network; and
providing the trained encoder neural network and the generator neural network for generating an output image for display.

8. The computing system of claim 7 , wherein the loss between the generated training image and the input training image is weighted by a first value, and the loss between the synthetic training code and the sampled latent code is weighted by a second value different from the first value.

9. The computing system of claim 7 , the processing further comprising sampling in a neighborhood of the transformed initial latent code to generate the sampled latent code.

10. The computing system of claim 7 , the processing further comprising:
providing a second input training image as input to the encoder neural network to produce a latent space representation of the second input training image;
sampling a second latent code from the latent space by sampling in a neighborhood of the latent space representation of the second input training image; and
retraining the encoder neural network using the second sampled latent code.

11. The computing system of claim 10 , the processing further comprising:
providing a third input training image as input to the encoder neural network to produce a latent space representation of the third input training image;
generating a third generated training image by providing the latent space representation of the third input training image as input to the generator neural network;
minimizing a loss between the latent space representation of the third input training image and the third generated training image to optimize the latent space representation of the third input training image;
sampling a third latent code from the latent space by sampling in a neighborhood of the optimized latent space representation of the third input training image; and
retraining the encoder neural network using the third sampled latent code.

12. The computing system of claim 11 , wherein retraining the encoder neural netwo

Assignees

• Adobe Inc

Inventors

• Kalarot Ratheesh
• Lin Wei-An
• Smith Cameron
• Shu Zhixin
• Faieta Baldo
• Ghadar Shabnam
• Lu Jingwan
• Darabi Aliakbar
• Zhu Jun-Yan
• Mitra Niloy
• Zhang Richard
• Shechtman Elya

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/094 Primary

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title