Systems and methods for generating personalized emoticons and lip synching videos based on facial recognition

Patent metadata

Field	Value
Publication number	US-2019206441-A1
Application number	US-201715857098-A
Country	US
Kind code	A1
Filing date	Dec 28, 2017
Priority date	Dec 28, 2017
Publication date	Jul 4, 2019
Grant date	—

Abstract

Systems, methods, and non-transitory computer readable media can obtain a first image of a first user depicting a face of the first user with a neutral expression or position. A first image of a second user depicting a face of the second user with a neutral expression or position can be identified, wherein the face of the second user is similar to the face of the first user based on satisfaction of a threshold value. A second image of the first user depicting the face of the first user with an expression different from the neutral expression or position can be generated based on a second image of the second user depicting the face of the second user with an expression or position different from the neutral expression or position.

First claim

1 . A computer-implemented method comprising:
obtaining, by a computing system, a first image of a first user depicting a face of the first user with a neutral expression or position; identifying, by the computing system, a first image of a second user depicting a face of the second user with a neutral expression or position, wherein the face of the second user is similar to the face of the first user based on satisfaction of a threshold value; and generating, by the computing system, a second image of the first user depicting the face of the first user with an expression different from the neutral expression or position, by mapping the facial points in the first image of the first user to the facial points in a second image of the second user depicting the face of the second user with an expression or position different from the neutral expression or position.

2 . The computer-implemented method of claim 1 , wherein the identifying the first image of the second user includes comparing facial points of the first user in the first image of the first user and facial points of the second user in the first image of the second user, wherein a degree of a match between the facial points of the first user and the facial points of the second user satisfies the threshold value.

3 . (canceled)

4 . The computer-implemented method of claim 1 , further comprising generating a personalized emoticon for the first user based on the second image of the first user.

5 . The computer-implemented method of claim 1 , wherein the first image of the second user is included in a video of the second user singing a song, and wherein the method further comprises generating a lip synching video of the first user based on the second image of the first user.

6 . The computer-implemented method of claim 1 , wherein the identifying the first image of the second user includes comparing a texture of at least a region in the first image of the first user and a texture of a corresponding region in the first image of the second user, wherein a degree of a match between the texture of the at least a region in the first image of the first user and the texture of the at least a region in the first image of the second user satisfies a threshold value.

7 . The computer-implemented method of claim 6 , wherein a portion of the at least a region in the first image of the second user is copied to generate the second image of the first user.

8 . The computer-implemented method of claim 6 , wherein the comparing the texture of the at least a region in the first image of the first user and the texture of the corresponding region in the first image of the second user includes determining texture features of the at least a region in the first image of the first user and texture features of the at least a region in the first image of the second user.

9 . The computer-implemented method of claim 8 , wherein an image is represented as a matrix of values, and texture features of the image includes one or more of: an average of the values or a median of the values.

10 . The computer-implemented method of claim 1 , wherein the expression different from the neutral expression includes one or more of: a happy expression, a sad expression, an angry expression, a surprise expression, a crying expression, a smiling expression, a laughing expression, or a frowning expression.

11 . A system comprising:
at least one hardware processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
obtaining a first image of a first user depicting a face of the first user with a neutral expression or position;
identifying a first image of a second user depicting a face of the second user with a neutral expression or position, wherein the face of the second user is similar to the face of the first user based on satisfaction of a threshold value; and
generating a second image of the first user depicting the face of the first user with an expression different from the neutral expression or position, by mapping the facial points in the first image of the first user to the facial points in a second image of the second user depicting the face of the second user with an expression or position different from the neutral expression or position.

12 . The system of claim 11 , wherein the identifying the first image of the second user includes comparing facial points of the first user in the first image of the first user and facial points of the second user in the first image of the second user, wherein a degree of a match between the facial points of the first user and the facial points of the second user satisfies the threshold value.

13 . (canceled)

14 . The system of claim 11 , further comprising generating a personalized emoticon for the first user based on the second image of the first user.

15 . The system of claim 11 , wherein the first image of the second user is included in a video of the second user singing a song, and wherein the instructions further cause the system to perform generating a lip synching video of the first user based on the second image of the first user.

16 . A non-transitory computer readable medium including instructions that, when executed by at least one hardware processor of a computing system, cause the computing system to perform a method comprising:
obtaining a first image of a first user depicting a face of the first user with a neutral expression or position; identifying a first image of a second user depicting a face of the second user with a neutral expression or position, wherein the face of the second user is similar to the face of the first user based on satisfaction of a threshold value; and generating a second image of the first user depicting the face of the first user with an expression different from the neutral expression or position, by mapping the facial points in the first image of the first user to the facial points in a second image of the second user depicting the face of the second user with an expression or position different from the neutral expression or position.

17 . The non-transitory computer readable medium of claim 16 , wherein the identifying the first image of the second user includes comparing facial points of the first user in the first image of the first user and facial points of the second user in the first image of the second user, wherein a degree of a match between the facial points of the first user and the facial points of the second user satisfies the threshold value.

18 . (canceled)

19 . The non-transitory computer readable medium of claim 16 , wherein the method further comprises generating a personalized emoticon for the first user based on the second image of the first user.

20 . The non-transitory computer readable medium of claim 16 , wherein the first image of the second user is included in a video of the second user singing a song, and wherein the method further comprises generating a lip synching video of the first user based on the second image of the first user.

Assignees

• Facebook Inc

Inventors

• De La Torre Fernando
• Huang Dong
• Vicente Carrasco Francisco

Classifications

• G06V40/172 Primary

  Classification, e.g. identification · CPC title

• G06V10/758

  Involving statistics of pixels or of feature values, e.g. histogram matching · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V40/175 Primary

  Static expression · CPC title

• G11B27/036 Primary

  Insert-editing · CPC title