Systems and methods for dynamically generating emojis based on image analysis of facial features

Patent metadata

Field	Value
Publication number	US-10025972-B2
Application number	US-201514942784-A
Country	US
Kind code	B2
Filing date	Nov 16, 2015
Priority date	Nov 16, 2015
Publication date	Jul 17, 2018
Grant date	Jul 17, 2018

Abstract

Systems, methods, and non-transitory computer-readable media can acquire real-time image data depicting at least a portion of a face of a user of a computing system (or device). The real-time image data can be analyzed to determine a state associated with at least the portion of the face. An emoji can be provided based on the state associated with at least the portion of the face. The emoji can be inputted in a communication to be made by the user.

First claim

What is claimed is:

1. A computer-implemented method comprising:
acquiring, by a computing system, real-time image data depicting at least a portion of a face of a user of the computing system;
analyzing, by the computing system, the real-time image data to determine a state associated with at least the portion of the face;
providing, by the computing system, an emoji based on the state associated with at least the portion of the face; and
inputting, by the computing system, the emoji based on the state associated with at least the portion of the face in a communication to be made by the user.

2. The computer-implemented method of claim 1 , further comprising:
analyzing the real-time image data to determine one or more positions of one or more specified facial features within at least the portion of the face of the user;
identifying one or more emoji features within the emoji that are associated with the one or more specified facial features; and
modifying the one or more emoji features based on the one or more positions of the one or more specified facial features.

3. The computer-implemented method of claim 2 , wherein the one or more specified facial features include at least one of an eye of the user, an eyebrow of the user, a nose of the user, a lip of the user, a tooth of the user, a tongue of the user, or a mouth of the user.

4. The computer-implemented method of claim 1 , further comprising:
analyzing the real-time image data to determine that the real-time image data further depicts at least one of an object or a gesture;
selecting a graphical representation for the at least one of the object or the gesture; and
providing the graphical representation in conjunction with the emoji.

5. The computer-implemented method of claim 1 , wherein analyzing the real-time image data to determine the state associated with at least the portion of the face further comprises:
analyzing one or more virtual points on at least the portion of the face to identify a virtual point arrangement; and
matching, within an allowable deviation, the virtual point arrangement with a facial expression model out of a plurality of facial expression models.

6. The computer-implemented method of claim 5 , further comprising:
identifying the emoji out of a plurality of emojis based on the facial expression model, wherein each of the plurality of emojis is respectively associated with each of the plurality of facial expression models.

7. The computer-implemented method of claim 5 , wherein matching the virtual point arrangement with the facial expression model is based on at least one of a machine learning training process or a crowdsource training process.

8. The computer-implemented method of claim 1 , further comprising:
analyzing the real-time image data to determine a second state associated with at least the portion of the face; and
updating, based on the second state associated with at least the portion of the face, the emoji provided.

9. The computer-implemented method of claim 1 , further comprising:
publishing the communication as at least one of a message, a journal entry, a diary entry, a blog entry, a comment, a response, or a post.

10. The computer-implemented method of claim 1 , further comprising:
dynamically presenting a preview of the emoji prior to inputting the emoji in the communication to be made by the user.

11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
acquiring real-time image data depicting at least a portion of a face of a user of the system;
analyzing the real-time image data to determine a state associated with at least the portion of the face;
providing an emoji based on the state associated with at least the portion of the face; and
inputting the emoji based on the state associated with at least the portion of the face in a communication to be made by the user.

12. The system of claim 11 , wherein the instructions cause the system to further perform:
analyzing the real-time image data to determine one or more positions of one or more specified facial features within at least the portion of the face of the user;
identifying one or more emoji features within the emoji that are associated with the one or more specified facial features; and
modifying the one or more emoji features based on the one or more positions of the one or more specified facial features.

13. The system of claim 12 , wherein the one or more specified facial features include at least one of an eye of the user, an eyebrow of the user, a nose of the user, a lip of the user, a tooth of the user, a tongue of the user, or a mouth of the user.

14. The system of claim 11 , wherein the instructions cause the system to further perform:
analyzing the real-time image data to determine that the real-time image data further depicts at least one of an object or a gesture;
selecting a graphical representation for the at least one of the object or the gesture; and
providing the graphical representation in conjunction with the emoji.

15. The system of claim 11 , wherein analyzing the real-time image data to determine the state associated with at least the portion of the face further comprises:
analyzing one or more virtual points on at least the portion of the face to identify a virtual point arrangement; and
matching, within an allowable deviation, the virtual point arrangement with a facial expression model out of a plurality of facial expression models.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
acquiring real-time image data depicting at least a portion of a face of a user of the computing system;
analyzing the real-time image data to determine a state associated with at least the portion of the face;
providing an emoji based on the state associated with at least the portion of the face; and
inputting the emoji based on the state associated with at least the portion of the face in a communication to be made by the user.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions cause the computing system to further perform:
analyzing the real-time image data to determine one or more positions of one or more specified facial features within at least the portion of the face of the user;
identifying one or more emoji features within the emoji that are associated with the one or more specified facial features; and
modifying the one or more emoji features based on the one or more positions of the one or more specified facial features.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the one or more specified facial features include at least one of an eye of the user, an eyebrow of the user, a nose of the user, a lip of the user, a tooth of the user, a tongue of the user, or a mouth of the user.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions cause the computing system to further perform:
analyzing the real-time image data to determine that the real-time image data further depicts at least one of an object or a gesture;
selecting a graphical representation for the at least one of the object or the gesture; and
providing the graphical representation in conjunction with the emoji.

20. The non-transitory computer-readable

Assignees

• Facebook Inc

Inventors

• Matas Michael James
• Reckhow Michael Waldman
• Taigman Yaniv

Classifications

• G06V40/174 Primary

  Facial expression recognition · CPC title

• G06V40/176

  Dynamic expression · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06F40/106

  Display of layout of documents; Previewing · CPC title

• G06F40/166

  Editing, e.g. inserting or deleting · CPC title