Dynamic mask application

Patent metadata

Field	Value
Publication number	US-10636175-B2
Application number	US-201615388846-A
Country	US
Kind code	B2
Filing date	Dec 22, 2016
Priority date	Dec 22, 2016
Publication date	Apr 28, 2020
Grant date	Apr 28, 2020

Abstract

In one embodiment, a method includes identifying an emotion associated with an identified first object in one or more input images, selecting, based on the emotion, a mask from a set of masks, where the mask specifies one or more mask effects, and for each of the input images, applying the mask to the input image. Applying the mask includes generating graphical features based on the identified first object or a second object in the input images according to instructions specified by the mask effects, and incorporating the graphical features into an output image. The emotion may be identified based on graphical features of the identified first object. The graphical features of the identified object may include facial features. The selected mask may be selected from a lookup table that maps the identified emotion to the selected mask.

First claim

What is claimed is:

1. A method comprising:
by a computer system, identifying an expression associated with an identified first object in an input image, wherein the identified first object has one or more attributes;
by the computer system, accessing social data of a first user associated with the input image, wherein social data comprises information from a social graph of an online social network;
by the computer system, selecting, based on the identified expression and the social data of the first user, a mask from a plurality of masks, wherein the plurality of masks comprise masks previously selected by friends of the first user within the online social network, wherein the mask specifies one or more mask effects and displays a stylized expression corresponding to the identified expression; and
by the computer system, applying the mask to the input image, wherein applying the mask comprises:
generating one or more graphical features based at least in part on the stylized expression reflective of the expression associated with the identified first object according to instructions specified by the mask effects; and
incorporating the one or more graphical features into the input images to generate an output image, wherein the graphical features are incorporated based on the one or more attributes of the first object.

2. The method of claim 1 , wherein the expression is identified based on one or more graphical features of the identified first object.

3. The method of claim 2 , wherein the graphical features of the identified object comprise one or more facial features.

4. The method of claim 1 , wherein the selected mask is selected from a lookup table that maps the identified expression to the selected mask to display the stylized expression reflective of the identified expression.

5. The method of claim 1 , wherein the mask is selected from the plurality of masks based further on one or more user-related factors associated with the first user to display the stylized expression reflective of the expression associated with the identified first object.

6. The method of claim 1 , wherein the mask is selected from the plurality of masks when a selection condition associated with the mask is satisfied by the social data of the first user.

7. The method of claim 6 , wherein the selection condition is satisfied when, for an attribute associated with the social data, a value of the attribute associated with the mask matches a value of the attribute associated with the social data.

8. The method of claim 1 , wherein one or more expressions are associated with the mask, and the mask is selected when the identified expression corresponds to at least one of the expressions associated with the mask in order to display the stylized expression reflective of the expression associated with the identified first object.

9. The method of claim 8 , wherein the identifying an expression generates a confidence value for the identified expression,
wherein each of the one or more expressions associated with the mask is further associated with a threshold confidence condition, and the mask is selected when the confidence value satisfies the threshold confidence condition.

10. The method of claim 1 , wherein the plurality of masks comprises a first mask that generates one or more first features associated with a first expression of a person or character and a second mask that generates one or more second features associated with a second expression of the person or character,
wherein the first mask is selected when the identified expression is the first expression, and the second mask is selected when the identified expression is the second expression.

11. The method of claim 1 , wherein the mask effects include a mask image, and the graphical features incorporated into the output image include one or more features from the mask image.

12. The method of claim 11 , wherein the features from the mask image are scaled based on the size of the first or second object.

13. The method of claim 11 , wherein the features from the mask image are positioned in the output image at locations that are determined based on locations of corresponding features of the first or second object in the input images according to mask geometry information associated with the mask effects.

14. The method of claim 1 , wherein the graphical features generated by the mask effects to display the stylized expression reflective of the expression associated with the identified first object comprise one or more of: exaggerated facial features, renderings of inanimate physical objects, or stylized facial features.

15. The method of claim 1 , wherein the identified first object in the input image comprises an image of a human, human face, animal, animal face, or inanimate object.

16. The method of claim 1 , wherein the input image is captured using a camera.

17. The method of claim 1 , wherein the input image is a photograph.

18. The method of claim 1 , wherein the input image is a video.

19. The method of claim 1 , further comprising:
by the computing system, determining a geographic location associated with the input image, wherein selecting the mask from the plurality of masks is further based on the geographic location.

20. The method of claim 1 , further comprising:
by the computing system, transforming the mask by modifying one or more graphical features to reflect one or more changes in the one or more attributes and identified expression of the identified first object in a second image successive to the first image.

21. The method of claim 1 , wherein the social data comprises information related to the first user stored within an online social network.

22. The method of claim 1 , wherein the social data comprises a user profile, user interests, user preferences, user relationships on a social network, and location data.

23. The method of claim 1 , wherein the social data comprises attributes related to the first user through nodes and edges of a social graph database.

24. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
identify an expression associated with an identified first object in an input image, wherein the identified first object has one or more attributes;
access social data of a first user associated with the input image, wherein social data comprises information from a social graph of an online social network;
select, based on the identified expression and the social data of the first user, a mask from a plurality of masks, wherein the plurality of masks comprise masks previously selected by friends of the first user within the online social network, wherein the mask specifies one or more mask effects and displays a stylized expression corresponding to the identified expression; and
apply the mask to the input image, wherein to apply the mask, the software is further operable when executed to:
generate one or more graphical features based at least in part on the stylized expression reflective of the expression associated with the identified first object according to instructions specified by the mask effects; and
incorporate the one or more graphical features into the input images to generate an output image, wherein the graphical features are incorporated based on the attribute of the first object.

25. A system comprising: one or more processors; and a memory coupled to the proc

Assignees

• Facebook Inc

Inventors

• Caballero Maria Luz
• Fowler Molly Jane
• Lu Congxi
• Hodgkins Charles Joseph
• Cuellar Daniel Moreno

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06K9/00228

  Physics · mapped topic

• G06K9/00302

  Physics · mapped topic

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06V40/174

  Facial expression recognition · CPC title