Cascaded domain bridging for image generation

Patent metadata

Field	Value
Publication number	US-12299799-B2
Application number	US-202218046073-A
Country	US
Kind code	B2
Filing date	Oct 12, 2022
Priority date	Oct 12, 2022
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

A method of generating a stylized 3D avatar is provided. The method includes receiving an input image of a user, generating, using a generative adversarial network (GAN) generator, a stylized image, based on the input image, and providing the stylized image to a first model to generate a first plurality of parameters. The first plurality of parameters include a discrete parameter and a continuous parameter. The method further includes providing the stylized image and the first plurality of parameters to a second model that is trained to generate an avatar image, receiving, from the second model, the avatar image, comparing the stylized image to the avatar image, based on a loss function, to determine an error, updating the first model to generate a second plurality of parameters that correspond to the first plurality of parameters, based on the error, and providing the second plurality of parameters as an output.

First claim

What is claimed is:

1. A method of generating a stylized 3D avatar, the method comprising:
receiving an input image of a user;
generating, using a generative adversarial network (GAN) generator, a stylized image, based on the input image;
providing the stylized image to a first model to generate a first plurality of parameters, the first plurality of parameters comprising one or more discrete parameters and one or more continuous parameters, the one or more continuous parameters comprise head characteristics, mouth characteristics, nose characteristics, ear characteristics, and eye characteristics, each corresponding to the user, wherein:
the head characteristics comprise a head width, a head length, and a blend shape coefficient for a head shape,
the mouth characteristics comprise a mouth width, a mouth volume, and a mouth position,
the nose characteristics comprise a nose width, a nose height, and a nose position,
the eye characteristics comprise an eye size, an eye spacing, and an eye rotation, and
the ear characteristics comprise an ear size,

providing the stylized image and the first plurality of parameters to a second model, the second model being trained to generate an avatar image;
receiving, from the second model, the avatar image;
comparing the stylized image to the avatar image, based on a loss function, to determine an error;
updating the first model to generate a second plurality of parameters, based on the error, the second plurality of parameters corresponding to the first plurality of parameters; and
providing the second plurality of parameters as an output.

2. The method of claim 1 , wherein the avatar image is a first image, wherein the error is a first error, and wherein the method further comprises:
receiving a predetermined loss threshold corresponding to the loss function;
providing the stylized image and the second plurality of parameters to the second model to generate a second avatar image;
comparing the stylized image to the second avatar image, based on the loss function, to determine a second error; and
providing the second plurality of parameters as the output, in response to determining that the second error is less than the predetermined loss threshold.

3. The method of claim 1 , wherein the loss function is based on a color loss corresponding to a difference between the stylized image and the avatar image, with respect to the one or more discrete parameters.

4. The method of claim 3 , wherein the loss function is further based on an identity loss and a perception loss, wherein the identity loss corresponds to a difference in a global appearance between the avatar and stylized images, wherein the avatar image and the stylized image each comprise a respective plurality of pixels, and wherein the perception loss corresponds to a difference between each pixel of the plurality of pixels of the stylized image to each respective pixel of the plurality of pixels of the avatar image.

5. The method of claim 1 , wherein the one or more discrete parameters comprise one or more from the group of: hair type, brow type, beard type, glasses type, eyelash type, eye makeup type, eye color type, brow color type, skin tone, hair color type, beard color type, mouth color type, and glasses color type.

6. A system for generating a stylized 3D avatar, the system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the system to perform a set of operations, the set of operations comprising:
receiving an input image of a user;
generating, using a generative adversarial network (GAN) generator, a stylized image, based on the input image;
providing the stylized image to a first model to generate a first plurality of parameters, the first plurality of parameters comprising one or more discrete parameters and one or more continuous parameters, the one or more continuous parameters comprise head characteristics, mouth characteristics, nose characteristics, ear characteristics, and eye characteristics, each corresponding to the user, wherein:
the head characteristics comprise a head width, a head length, and a blend shape coefficient for a head shape,
the mouth characteristics comprise a mouth width, a mouth volume, and a mouth position,
the nose characteristics comprise a nose width, a nose height, and a nose position,
the eye characteristics comprise an eye size, an eye spacing, and an eye rotation, and
the ear characteristics comprise an ear size,

providing the stylized image and the first plurality of parameters to a second model, the second model being trained to generate an avatar image;
receiving, from the second model, the avatar image;
comparing the stylized image to the avatar image, based on a loss function, to determine an error;
updating the first model to generate a second plurality of parameters, based on the error, the second plurality of parameters corresponding to the first plurality of parameters; and
providing the second plurality of parameters as an output.

7. The system of claim 6 , wherein the avatar image is a first image, wherein the error is a first error, and wherein the set of operations further comprise:
receiving a predetermined loss threshold corresponding to the loss function;
providing the stylized image and the second plurality of parameters to the second model to generate a second avatar image;
comparing the stylized image to the second avatar image, based on the loss function, to determine a second error; and
providing the second plurality of parameters as the output, in response to determining that the second error is less than the predetermined loss threshold.

8. The system of claim 6 , wherein the loss function is based on a color loss corresponding to a difference between the stylized image and the avatar image, with respect to the one or more discrete parameters.

9. The system of claim 8 , wherein the loss function is further based on an identity loss and a perception loss, wherein the identity loss corresponds to a difference in a global appearance between the avatar and stylized images, wherein the avatar image and the stylized image each comprise a respective plurality of pixels, and wherein the perception loss corresponds to a difference between each pixel of the plurality of pixels of the stylized image to each respective pixel of the plurality of pixels of the avatar image.

10. The system of claim 6 , wherein the one or more discrete parameters comprise one or more from the group of: hair type, brow type, beard type, glasses type, eyelash type, eye makeup type, eye color type, brow color type, skin tone, hair color type, beard color type, mouth color type, and glasses color type.

11. A non-transient computer-readable storage medium comprising instructions being executable by one or more processors to cause the one or more processors to:
receive an input image of a user;
generate, using a generative adversarial network (GAN) generator, a stylized image, based on the input image;
provide the stylized image to a first model to generate a first plurality of parameters, the first plurality of parameters comprising one or more discrete parameters and one or more continuous parameters, the one or more continuous parameters comprise head characteristics, mouth characteristics, nose characteristics, ear characteristics, and eye characteristics, each corresponding to the user, wherein:
the head characteristics comprise a head width, a head length, and a blend shape coefficient for a head shape,
the mouth characteristics comprise a mouth width, a mouth volume, and a mouth position,
the nose characteristics comprise a nose width, a nose height, and a nose positi

Assignees

• Lemon Inc
• Beijing Zitiao Network Technology Co Ltd

Inventors

• Sang Shen
• Zhi Tiancheng
• Song Guoxian
• Liu Jing
• Luo Linjie
• Lai Chunpong
• Zeng Weihong
• Sun Jingna
• Wang Xu

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06T2207/10024

  Color image · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06T2207/30201

  Face · CPC title

• G06V10/56

  relating to colour · CPC title