Vectorized caricature avatar generator

Patent metadata

Field	Value
Publication number	US-11854115-B2
Application number	US-202117519117-A
Country	US
Kind code	B2
Filing date	Nov 4, 2021
Priority date	Nov 4, 2021
Publication date	Dec 26, 2023
Grant date	Dec 26, 2023

Abstract

A vectorized caricature avatar generator receives a user image from which face parameters are generated. Segments of the user image including certain facial features (e.g., hair, facial hair, eyeglasses) are also identified. Segment parameter values are also determined, the segment parameter values being those parameter values from a set of caricature avatars that correspond to the segments of the user image. The face parameter values and the segment parameter values are used to generate a caricature avatar of the user in the user image.

First claim

What is claimed is:

1. In a digital medium environment, a method implemented by at least one computing device, the method comprising:
receiving an image including a user face;
identifying a segment of the image that identifies a facial feature of the user face;
generating, for each of multiple face parameters, a face parameter value by processing a first portion of the user face that excludes the facial feature using a first machine learning system;
determining segment parameter values for the segment of the image by processing a second portion of the user face that includes the facial feature and excludes the first portion of the user face using a second machine learning system that is separately trained from the first machine learning system to determine the segment parameter values for the facial feature, the second machine learning system trained using training data that includes corresponding segments of multiple user images that identify the facial feature in the multiple user images and known segment parameter values for the corresponding segments of caricature avatars corresponding to the multiple user images, the second machine learning system having been trained using the known segment parameter values that are changed from default values during manual generation of the caricature avatars while masking the known segment parameter values that are unchanged from the default values during the manual generation of the caricature avatars; and
generating, from the face parameter values and the segment parameter values, a vector image that is a caricature avatar of a user depicted in the image.

2. The method of claim 1 , wherein the generating the face parameter value includes generating the face parameter value using the first machine learning system trained using training data that includes a facial texture of each of the multiple user images and known face parameter values corresponding to each of the multiple user images.

3. The method of claim 2 , the known face parameter values and the known segment parameter values having been generated manually by generating the caricature avatars for users depicted in the multiple user images.

4. The method of claim 2 , further comprising training the first machine learning system by:
receiving a user image of the multiple user images;
generating the facial texture of the first portion of an additional user face depicted in the user image that excludes the facial feature;
determining, using the first machine learning system, additional face parameter values of the user image based on the facial texture; and
updating the first machine learning system based on a loss between the additional face parameter values and the known face parameter values of the user image.

5. The method of claim 1 , the determining the segment parameter values including:
identifying, using the second machine learning system, intermediary segment parameter values for the segment of the image;
determining one of the caricature avatars having the known segment parameter values of a corresponding segment with a smallest difference to the intermediary segment parameter values of the segment of the image; and
determining, as the segment parameter values, the known segment parameter values of the determined one of the caricature avatars.

6. The method of claim 1 , wherein the training data includes a facial texture of the corresponding segments that identify the facial feature in the multiple user images and the known segment parameter values.

7. The method of claim 1 , further comprising:
receiving user input changing at least one of the face parameter values; and
generating, from the face parameter values including the at least one changed face parameter value, a new caricature avatar of the user.

8. The method of claim 1 , further comprising training the second machine learning system by:
receiving a user image of the multiple user images;
identifying a corresponding segment of the user image that identifies the facial feature;
generating one or more rotated versions of the corresponding segment of the user image;
generating, using the second machine learning system, additional segment parameter values from the one or more rotated versions of the corresponding segment; and
updating the second machine learning system based on a loss between the additional segment parameter values generated from the one or more rotated versions of the corresponding segment and the known segment parameter values for the corresponding segment of the user image.

9. The method of claim 1 , wherein the generating the face parameter value includes:
identifying dense key points of the image representing a geometry of the user face;
extracting a facial texture of the user face that is a set of vectors representing the user face; and
generating, for each of the multiple face parameters, the face parameter value based on the dense key points and the facial texture.

10. The method of claim 1 , further comprising:
receiving user input changing at least one of the segment parameter values; and
generating, from the segment parameters including the at least one changed segment parameter value, a new caricature avatar of the user.

11. The method of claim 1 , further comprising training the second machine learning system by:
receiving a user image of the multiple user images;
generating a facial texture of a corresponding segment of the user image that identifies the facial feature;
determining, using the second machine learning system, additional segment parameter values of the corresponding segment based on the facial texture; and
updating the second machine learning system based on a loss between the additional segment parameter values and the known segment parameter values for the corresponding segment of the user image.

12. The method of claim 1 , wherein the facial feature is hair, facial hair, or eyeglasses.

13. In a digital medium environment, a computing device comprising:
a processor; and
computer-readable storage media having stored thereon multiple instructions of an application that, responsive to execution by the processor, cause the processor to:
receive an image including a user face;
identify a segment of the image that identifies a facial feature of the user face;
generate, for each of multiple face parameters, a face parameter value by processing a first portion of the user face that excludes the facial feature using a first machine learning system;
identify, for the segment, segment parameter values by processing a second portion of the user face that includes the facial feature and excludes the first portion of the user face using a second machine learning system that is separately trained from the first machine learning system to identify the segment parameter values for the facial feature, the second machine learning system trained using training data that includes corresponding segments of multiple user images that identify the facial feature in the multiple user images and known segment parameter values for the corresponding segments of caricature avatars corresponding to the multiple user images, the second machine learning system having been trained using the known segment parameter values that are changed from default values during manual generation of the caricature avatars while masking the known segment parameter values that are unchanged from the default values during the manual generation of the caricature avatars; and
generate, from the face parameter values and the segment parameter values, a vector image that is a caricature avatar of a user depicted in the image.

14. The com

Assignees

• Adobe Inc

Inventors

• Ito Daichi
• Li Yijun
• Hold-Geoffroy Yannick
• Madono Koki
• Vallespi Jose Ignacio Echevarria
• Smith Cameron Younger

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T7/10

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20092

  Interactive image processing based on input by user · CPC title