Systems and methods for creating animations using human faces

Patent metadata

Field	Value
Publication number	US-9626788-B2
Application number	US-201615044970-A
Country	US
Kind code	B2
Filing date	Feb 16, 2016
Priority date	Mar 6, 2012
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

Systems and methods in accordance with embodiments of the invention enable collaborative creation, transmission, sharing, non-linear exploration, and modification of animated video messages. One embodiment includes a video camera, a processor, a network interface, and storage containing an animated message application, and a 3D character model. In addition, the animated message application configures the processor to: capture a video sequence using the video camera; detect a human face within a sequence of video frames; track changes in human facial expression of a human face detected within a sequence of video frames; map tracked changes in human facial expression to motion data, where the motion data is generated to animate the 3D character model; apply motion data to animate the 3D character model; render an animation of the 3D character model into a file as encoded video; and transmit the encoded video to a remote device via the network interface.

First claim

We claim:

1. A computer-implemented method for generating animations messages using video images of human faces, the method comprising:
receiving a sequence of video frames;
detecting a human face within the sequence of the video frames;
tracking, within the sequence of video frames, facial expression changes in the detected human face;
identifying, for each of the tracked facial expression changes, a facial expression from a training set of a plurality of different facial expressions and a plurality of different user faces that most closely corresponds to the tracked facial expression change;
mapping, by at least one processor, each of the identified facial expressions to three-dimensional motion data such that the identified facial expressions corresponds with the tracked facial expression changes from the sequence of video frames; and
applying the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model.

2. The method of claim 1 , further comprising detecting human facial expressions from the detected human face within the sequence of video frames.

3. The method of claim 2 , wherein identifying the facial expression further comprises matching the tracked facial expression change to a facial expression model from the training set of the plurality of different facial expressions and the plurality of different user faces that is stored in a database of facial expressions.

4. The method of claim 3 , further comprising obtaining, based on matching the tracked facial expression change to the facial expression model, the three-dimensional motion data for the facial expression model.

5. The method of claim 1 , wherein the three-dimensional character model comprises data that describes an articulated skeleton, at least one mesh, and skinning weights that indicate how the articulated skeleton deforms the at least one mesh for a given motion.

6. The method of claim 5 , wherein the articulated skeleton comprises a plurality of bones configured to rotate in one or more degrees of freedom around joints, and wherein the at least one mesh comprises a plurality of vertices defining a surface having a plurality of polygons.

7. The method of claim 1 , wherein detecting the human face within the sequence of the video frames comprises using a cascade of classifiers to detect the human face within the sequence of video frames.

8. The method of claim 1 , wherein identifying the facial expression further comprises:
obtaining a feature vector of the tracked facial expression change within a principal component analysis (PCA) space; and identifying, in the PCA space, the facial expression from the training set of the plurality of different facial expressions and the plurality of different user faces that has a feature vector that most closely matched the obtained feature vector of the tracked facial expression.

9. The method of claim 1 , wherein tracking facial expression changes comprises tracking landmark points, edges, and colors of the detected human face within the sequence of the video frames.

10. The method of claim 1 , wherein identifying the facial expression further comprises performing principal component analysis of the training set of the plurality of different facial expressions and the plurality of different user faces to identify the facial expression that most closely corresponds to the tracked facial expression change.

11. The method of claim 1 , wherein the training set of the plurality of different facial expressions and the plurality of different user faces comprise a set of three-dimensional images that include additional metadata including depth maps.

12. The method of claim 1 , further comprising animating the face of the three-dimensional character model by deforming the mesh of the three-dimensional character using a sequence of facial expressions for the three-dimensional character model based on the applied three-dimensional motion data.

13. A computer-implemented method for generating animations using video images of human faces, the method comprising:
capturing a sequence of video frames;
detecting a facial expression within the sequence of the video frames;
matching, by at least one processor, the detected facial expression to a stored facial expression from a training set of a plurality of different facial expressions and a plurality of different user faces that most closely corresponds to the detected facial expression;
obtaining three-dimensional motion data associated with the stored facial expression that matches the detected facial expression; and
applying the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model such that the animation replicates the detected facial expression on the three-dimensional character model.

14. The method of claim 13 , wherein the training set of the plurality of different facial expressions and the plurality of different user faces is stored in a database of stored facial expressions and comprises a set of three-dimensional images that include additional metadata including depth maps.

15. The method of claim 14 , wherein matching the detected facial expression to the stored facial expression comprises matching the detected facial expression to the stored facial expression from the training set of the plurality of different facial expressions and the plurality of different user faces by comparing feature vectors between the detected facial expression and facial expressions in the training set of the plurality of different facial expressions and the plurality of different user faces.

16. The method of claim 14 , wherein matching the detected facial expression to the stored facial expression comprises further performing principal component analysis of the training set of the plurality of different facial expressions and the plurality of different user faces to identify the facial expression that most closely corresponds to the detected facial expression.

17. The method of claim 14 , further comprising locating a vector within the training set of the plurality of different facial expressions and the plurality of different user faces that is the closest match to a vector representing the detected face.

18. A system for generating animations using video images of human faces, the system comprising:
at least one processor; and
at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to:
receive a sequence of video frames;
detect a human face within the sequence of the video frames;
track, within the sequence of video frames, facial expression changes in the detected human face;
identify, for each of the tracked facial expression changes, a facial expression from a training set of a plurality of different facial expressions and a plurality of different user faces that most closely corresponds to the tracked facial expression change;
map each identified facial expression to three-dimensional motion data such that the identified facial expressions corresponds with the tracked facial expression changes from the sequence of video frames; and
apply the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model.

19. The system of claim 18 , wherein the instructions, when executed by the at least one processor, cause the system to identify a facial expression by:
obtaining a feature vector o

Assignees

• Adobe Systems Inc

Inventors

• Corazza Stefano
• Babcock Daniel J
• Pina Charles
• Drouin Sylvio

Classifications

• G06V40/176

  Dynamic expression · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06K9/00228

  Physics · mapped topic

• G06K9/00315

  Physics · mapped topic