Generating cartoon images from photos

Patent metadata

Field	Value
Publication number	US-10853987-B2
Application number	US-201916702440-A
Country	US
Kind code	B2
Filing date	Dec 3, 2019
Priority date	Mar 20, 2017
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

A system and method for generating cartoon images from photos are described. The method includes receiving an image of a user, determining a template for a cartoon avatar, determining an attribute needed for the template, processing the image with a classifier trained for classifying the attribute included in the image, determining a label generated by the classifier for the attribute, determining a cartoon asset for the attribute based on the label, and rendering the cartoon avatar personifying the user using the cartoon asset.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, using one or more computing devices, an image of a user;
determining, using the one or more computing devices, a template for an avatar;
determining, based on the image of the user and using the one or more computing devices, attributes needed for the template, wherein the attributes include a plurality of facial attributes;
processing, using the one or more computing devices, the image with a respective classifier trained to generate a corresponding label that corresponds to a value of each of the attributes, wherein each classifier is coupled to a selected layer of a pre-trained neural network model and wherein the selected layer of the pre-trained neural network model is selected by determining that the selected layer provides discrimination for a corresponding attribute;
determining, using the one or more computing devices, a set of assets for the avatar based on labels; and
rendering, using the one or more computing devices, the avatar personifying the user using the set of assets by:
organizing each asset of the set of assets into a position relative to other assets corresponding to the plurality of facial attributes in the template; and
rendering the avatar based on the organized assets.

2. The computer-implemented method of claim 1 , further comprising dividing the attributes into a primary attribute list and a secondary attribute list.

3. The computer-implemented method of claim 2 , wherein determining the set of assets based on the labels is based on ranking candidate assets associated with each label based on a level of match between each candidate asset and the label, and using the candidate assets that meet a threshold rank.

4. The computer-implemented method of claim 1 , wherein the set of assets are a first set of assets that correspond to a first style and are associated with a face parameterization and further comprising:
receiving a second set of assets that correspond to a second style or emotional expression;
determining whether the second style or emotional expression is compatible with the face parameterization; and
responsive to determining that the second style or the emotional expression is compatible with the face parameterization, determining a new set of assets for the second style or emotional expression.

5. The computer-implemented method of claim 4 , wherein determining the new set of assets includes:
mapping the first set of assets to the second style or emotional expression; and
deriving the new set of assets based on the mapping.

6. The computer-implemented method of claim 1 , wherein rendering the avatar further includes adding fixed elements that are specific to a style for the avatar.

7. The computer-implemented method of claim 1 , further comprising:
presenting the avatar on a display of a user device;
receiving user input on the display that is a manual edit of the avatar; and
rerendering the avatar based on a changed face parameterization, wherein the changed face parameterization is based on the user input.

8. A computer program product comprising a non-transitory computer readable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to perform operations comprising:
receiving, using one or more computing devices, an image of a user;
determining, using the one or more computing devices, a template for an avatar;
determining, based on the image of the user and using the one or more computing devices, attributes needed for the template, wherein the attributes include a plurality of facial attributes;
processing, using the one or more computing devices, the image with a respective classifier trained to generate a corresponding label that corresponds to a value of each of the attributes, wherein each classifier is coupled to a selected layer of a pre-trained neural network model and wherein the selected layer of the pre-trained neural network model is selected by determining that the selected layer provides discrimination for a corresponding attribute;
determining, using the one or more computing devices, a set of assets for the avatar based on labels; and
rendering, using the one or more computing devices, the avatar personifying the user using the set of assets by:
organizing each asset of the set of assets into a position relative to other assets corresponding to the plurality of facial attributes in the template; and
rendering the avatar based on the organized assets.

9. The computer program product of claim 8 , wherein the operations further comprise dividing the attributes into a primary attribute list and a secondary attribute list.

10. The computer program product of claim 9 , wherein determining the set of assets based on the labels is based on ranking candidate assets associated with each label based on a level of match between each candidate asset and the label, and using the candidate assets that meet a threshold rank.

11. The computer program product of claim 8 , wherein the set of assets are a first set of assets that correspond to a first style and are associated with a face parameterization and the operations further comprise:
receiving a second set of assets that correspond to a second style or emotional expression;
determining whether the second style or emotional expression is compatible with the face parameterization; and
responsive to determining that the second style or the emotional expression is compatible with the face parameterization, determining a new set of assets for the second style or emotional expression.

12. The computer program product of claim 11 , wherein determining the new set of assets includes:
mapping the first set of assets to the second style or emotional expression; and
deriving the new set of assets based on the mapping.

13. The computer program product of claim 8 , wherein rendering the avatar further includes adding fixed elements that are specific to a style for the avatar.

14. The computer program product of claim 8 , wherein the operations further comprise:
presenting the avatar on a display of a user device;
receiving user input on the display that is a manual edit of the avatar; and
rerendering the avatar based on a changed face parameterization, wherein the changed face parameterization is based on the user input.

15. A system comprising:
a processor; and
a memory coupled to the processor and storing instructions that, when executed, cause the processor to perform operations comprising:
receiving, using one or more computing devices, an image of a user;
determining, using the one or more computing devices, a template for an avatar;
determining, based on the image of the user and using the one or more computing devices, attributes needed for the template, wherein the attributes include a plurality of facial attributes;
processing, using the one or more computing devices, the image with a respective classifier trained to generate a corresponding label that corresponds to a value of each of the attributes, wherein each classifier is coupled to a selected layer of a pre-trained neural network model and wherein the selected layer of the pre-trained neural network model is selected by determining that the selected layer provides discrimination for a corresponding attribute;
determining, using the one or more computing devices, a set of assets for the avatar based on labels; and
rendering, using the one or more computing devices, the avatar personifying the user using the set of assets by:
organizing each asset of the set of a

Assignees

• Google Llc

Inventors

• Sarna Aaron
• Krishnan Dilip
• Cole Forrester
• Mosseri Inbar

Classifications

• G06T13/80 Primary

  Two-dimensional [2D] animation, e.g. using sprites · CPC title