Facial Expression Capture for Character Animation

Patent metadata

Field	Value
Publication number	US-2016275341-A1
Application number	US-201514661788-A
Country	US
Kind code	A1
Filing date	Mar 18, 2015
Priority date	Mar 18, 2015
Publication date	Sep 22, 2016
Grant date	—

Abstract

Techniques for facial expression capture for character animation are described. In one or more implementations, facial key points are identified in a series of images. Each image, in the series of images, is normalized from the identified facial key points. Facial features are determined from each of the normalized images. Then a facial expression is classified, based on the determined facial features, for each of the normalized images. In additional implementations, a series of images are captured that include performances of one or more facial expressions. The facial expressions in each image of the series of images are classified by a facial expression classifier. Then the facial expression classifications are used by a character animator system to produce a series of animated images of an animated character that include animated facial expressions that are associated with the facial expression classification of the corresponding image in the series of images.

First claim

What is claimed is:

1 . A computer-implemented method for classifying facial expressions in a series of images, the method comprising:
identifying facial key points in the series of images; normalizing each of the images using the identified facial key points for each of the images; determining facial features from each of the normalized images; and based on the determining the facial features, classifying a facial expression in each of the images into one of a plurality of categories.

2 . The method of claim 1 , further comprising temporally smoothing the classified facial expressions to reduce jitter between the categories of the classified facial expressions associated with the series of images.

3 . The method of claim 1 , wherein the determining the facial features comprises:
extracting geometric features from each of the images, the geometric features describing spatial deformations of the facial key points; and extracting appearance features from each of the images, the appearance features describing appearance changes due to the spatial deformation of the facial key points.

4 . The method of claim 3 , wherein the geometric features comprise a plurality of measurements based on shapes and locations of the facial features.

5 . The method of claim 3 , wherein the extracting the appearance features comprises:
partitioning each image into a uniform grid of patches; combining adjacent partitioned patches into a plurality of regions in each image; determining Histogram of Gradient (HoG) features for each of the regions; and concatenating the determined HoG features into an integrated vector.

6 . The method of claim 1 , the method further comprising:
dimensionally reducing the determined facial features; inputting the normalized images to a deep convoluted neural network (CNN), the CNN comprising a plurality of convolutional layers and max pooling layers; determining, by the CNN, additional facial features in the normalized images; and fusing the dimensionally reduced facial features with the determined additional facial features before the classifying the facial expression in each of the images.

7 . The method of claim 6 , wherein the determining the additional facial features is to determine the facial features associated with a plurality of canonical facial expressions, and wherein the determining the facial features from each of the normalized images is to determine the facial features associated with customized facial expressions.

8 . A system for character animation, the system comprising:
a facial expression classifier configured to classify one or more facial expressions of a user from a series of images that includes performances of the one or more facial expressions by the user, the facial expression classifier is configured to:
identify facial key points in each image of the series of images;
normalize each image of the series of images using the identified facial key points for each image;
determine facial features from each normalized image; and
based on the determined facial features, classify the facial expression of each image of the series of images into one of a plurality of categories; and
a character animator configured to produce a series of animated images of an animated character, the animated images of the animated character including one or more animated facial expressions, each animated facial expression being associated with a corresponding performed facial expression, classified by the facial expression classifier.

9 . The system of claim 8 , wherein a change of the classification of the one or more facial expressions from a first classification to a second classification is effective to enable the character animator to change artwork for the animation based on the change of the classification.

10 . The system of claim 8 , wherein to determine the facial features, the facial expression classifier is configured to:
extract geometric features from each image of the series of images, the geometric features describing spatial deformations of the facial key points; and extract appearance features from each image of the series of images, the appearance features describing appearance changes due to the spatial deformation of the facial key points.

11 . The system of claim 10 , wherein the geometric features comprise a plurality of measurements based on shapes and locations of the facial features.

12 . The system of claim 10 , wherein to extract of the appearance features, the facial expression classifier is further configured to:
partition each image of the series of images into a uniform grid of patches; combine adjacent partitioned patches into a plurality of regions in each image; determine Histogram of Gradient (HoG) features for each of the regions; and concatenate the determined HoG features into an integrated vector.

13 . The system of claim 8 , the facial expression classifier further configured to:
dimensionally reduce the determined facial features; input the normalized images to a deep convoluted neural network (CNN), the CNN comprising a plurality of convolutional layers and max pooling layers; determine, by the CNN, additional facial features in the normalized images; and fuse the dimensionally reduced facial features with the recognized additional facial features before the classification of the facial expression of each image.

14 . The system of claim 13 , wherein the determination of additional facial features is to determine the facial features associated with a plurality of canonical facial expressions and wherein the determination of facial features from each normalized image is to determine the facial features associated with customized facial expressions.

15 . The system of claim 8 , the facial expression classifier being further configured to:
temporally smooth the classified facial expressions to reduce jitter between character expressions provided to the character animator.

16 . The system of claim 8 further comprising:
an image capture device configured to capture the series of images for facial expression classification.

17 . A system for classifying facial expressions, the system comprising:
memory configured to store a plurality of images; a processing system to implement a facial expression classifier module as executable instructions configured to:
identify facial key points in the plurality of images;
normalize each of the images using the identified facial key points for each of the images;
determine facial features from each of the normalized images; and
based on the determined facial features, classify a facial expression in each of the images into one of a plurality of categories.

18 . The system of claim 17 , wherein the determined facial features comprise geometric features, which describe spatial deformations of the facial key points, and appearance features, which describe appearance changes due to the spatial deformation of the facial key points, and the facial expression classifier module configured to:
extract the geometric features from each of the images, the geometric features comprising a plurality of measurements based on shapes and locations of the facial features; and extract appearance features from each of the images, wherein to extract the appearance features the facial expression classifier module is further configured to:
partition each image into a uniform grid of patches;
combine adjacent partitioned patches into a plurality of regions in each image;
determine Histogram of Gradien

Assignees

• Adobe Systems Inc

Inventors

• Li Wilmot Wei-Mau
• Yang Jianchao
• Luo Linjie
• Brandt Jonathan W
• Yu Xiang

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06F18/24137

  Distances to cluster centroïds · CPC title