Portrait stylization framework to control the similarity between stylized portraits and original photo

Patent metadata

Field	Value
Publication number	US-12217466-B2
Application number	US-202117519711-A
Country	US
Kind code	B2
Filing date	Nov 5, 2021
Priority date	Nov 5, 2021
Publication date	Feb 4, 2025
Grant date	Feb 4, 2025

Abstract

Systems and methods directed to controlling the similarity between stylized portraits and an original photo are described. In examples, an input image is received and encoded using a variational autoencoder to generate a latent vector. The latent vector may be blended with latent vectors that best represent a face in the original user portrait image. The resulting blended latent vector may be provided to a generative adversarial network (GAN) generator to generate a controlled stylized image. In examples, one or more layers of the stylized GAN generator may be swapped with one or more layers of the original GAN generator. Accordingly, a user can interactively determine how much stylization vs. personalization should be included in a resulting stylized portrait.

First claim

What is claimed is:

1. A method for generating a stylized image, the method comprising:
receiving an input image;
generating, using a first encoder, a first latent code based on the input image;
generating, using a second encoder, a second latent code based on the input image;
causing a user interface to be displayed;
receiving, via input to the user interface, a blending parameter indicating an amount to blend the first latent code with the second latent code;
blending the first latent code and the second latent code to obtain a blended latent code;
generating, by a generative adversarial network (GAN) generator, a stylized image based on the blended latent code; and
providing the stylized image as an output.

2. The method of claim 1 , further comprising:
receiving a blending parameter indicating one or more layers of a first pre-trained GAN generator are to be used in the GAN generator;
assembling the GAN generator based on the blending parameter and the one or more layers of the pre-trained GAN generator; and
generating the stylized image using the assembled GAN generator.

3. The method of claim 2 , wherein the GAN generator is a trained GAN generator trained via transfer learning from the first pre-trained GAN generator.

4. The method of claim 1 , wherein the first encoder is a PSP encoder.

5. The method of claim 1 , wherein the second encoder is a variational hierarchical autoencoder.

6. The method of claim 1 , further comprising:
generating the first latent code from a first multilayer perceptron; and
generating the second latent code from a second multilayer perceptron.

7. A system, comprising:
one or more hardware processors configured by machine-readable instructions to:
receive an input image;
generate, using a first encoder, a first latent code based on the input image;
generate, using a second encoder, a second latent code based on the input image;
cause a user interface to be displayed;
receive, via input to the user interface, a blending parameter indicating an amount to blend the first latent code with the second latent code
blend the first latent code and the second latent code to obtain a blended latent code;
generate, by a generative adversarial network generator, a stylized image based on the blended latent code; and
provide the stylized image as an output.

8. The system of claim 7 , wherein the one or more hardware processors are further configured by machine-readable instructions to:
receive a blending parameter indicating one or more layers of a first pre-trained GAN generator are to be used in the GAN generator;
assemble the GAN generator based on the blending parameter and the one or more layers of the pre-trained GAN generator; and
generate the stylized image using the assembled GAN generator.

9. The system of claim 8 , wherein the GAN generator is a trained GAN generator trained via transfer learning from the first pre-trained GAN generator.

10. The system of claim 7 , wherein the first encoder is a PSP encoder.

11. The system of claim 7 , wherein the second encoder is a variational hierarchical autoencoder.

12. The system of claim 7 , wherein the one or more hardware processors are further configured by machine-readable instructions to:
generate the first latent code from a first multilayer perceptron; and
generate the second latent code from a second multilayer perceptron.

13. A non-transitory computer-readable storage medium comprising instructions, which when executed by one or more processors, cause the one or more processors to:
receive an input image;
generate, using a first encoder, a first latent code based on the input image;
generate, using a second encoder, a second latent code based on the input image;
cause a user interface to be displayed;
receive, via input to the user interface, a blending parameter indicating an amount to blend the first latent code with the second latent code
blend the first latent code and the second latent code to obtain a blended latent code;
generate, by a generative adversarial network generator, a stylized image based on the blended latent code; and
provide the stylized image as an output.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
receive a blending parameter indicating one or more layers of a first pre-trained GAN generator are to be used in the GAN generator;
assemble the GAN generator based on the blending parameter and the one or more layers of the pre-trained GAN generator; and
generate the stylized image using the assembled GAN generator.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the GAN generator is a trained GAN generator trained via transfer learning from the first pre-trained GAN generator.

16. The non-transitory computer-readable storage medium of claim 13 , wherein the second encoder is a variational hierarchical autoencoder.

17. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
generate the first latent code from a first multilayer perceptron; and
generate the second latent code from a second multilayer perceptron.

18. The method of claim 1 , wherein the causing a user interface to be displayed comprises causing a control to be displayed on the user interface, and wherein the control is interactable to allow a user to provide the blending parameter via the control.

19. The method of claim 18 , wherein the control is a slider.

20. The method of claim 18 , further comprising causing the stylized image to be displayed on the user interface.

Assignees

• Lemon Inc

Inventors

• Liu Jing
• Lai Chunpong
• Song Guoxian
• Luo Linjie

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title