Method for on-the-fly learning of facial artifacts for facial emotion recognition

Patent metadata

Field	Value
Publication number	US-9405962-B2
Application number	US-201313904225-A
Country	US
Kind code	B2
Filing date	May 29, 2013
Priority date	Aug 14, 2012
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

A method for determining a facial emotion of a user in the presence of a facial artifact includes detecting Action Units (AUs) for a first set of frames with the facial artifact; analyzing the AUs with the facial artifact after the detection; registering the analyzed AUs for a neutral facial expression with the facial artifact in the first set of frames; predicting the AUs in a second set of frames; and determining the facial emotion by comparing the registered neutral facial expression with the predicted AUs in the second set of frames.

First claim

What is claimed is:

1. A computer-implemented method for determining a facial emotion, in a video comprising a face with a facial artifact, the method comprising:
detecting Action Units (AUs) in a first set of frames of the video, wherein the first set of frames of the video comprises the face with the facial artifact, and wherein each AU comprises a fundamental action of an individual muscle or group of muscles;
analyzing the AUs of the first set of frames of the video, wherein analyzing the AUs comprises:
determining frequently occurring AUs that appear in a larger subset of a sequence of frames of the video than other AUs, and
registering the frequently occurring AUs as a neutral facial expression;

registering, as the neutral facial expression, the analyzed AUs in the first set of frames of the video;
predicting AUs in a second set of frames of the video; and
determining the facial emotion, by comparing: the AUs in the registered neutral facial expression in the first set of frames of the video, with the predicted AUs in the second set of frames of the video.

2. The method of claim 1 , wherein registering, as the neutral facial expression, the analyzed AUs comprises:
assuming that the face in the video shows the neutral facial expression in the first set of frames of the video.

3. The method of claim 1 , wherein detecting the AUs further comprises:
localizing the face in the video; and
extracting features of the localized face.

4. The method of claim 1 , wherein a weight of an AU with a facial artifact is reduced, if variations are detected in an AU of the second set of frames of the video and in the AU in the first set of frames of the video.

5. A non-transitory computer-readable recording medium storing a program that, when executed by a computer, implements the method of claim 1 .

6. A computing device for determining a facial emotion, in a video comprising a face with a facial artifact, the computing device comprising:
a processor configured to:
detect Action Units (AUs) in a first set of frames of the video, wherein the first set of frames of the video comprises the face with the facial artifact, and wherein each AU comprises a fundamental action of an individual muscle or group of muscles;
analyze the AUs to determine frequently occurring AUs that appear in a larger subset of the first set of frames of the video than other AUs; register, as a neutral facial expression, the frequently occurring AUs that appear in a larger subset of the first set of frames of the video than other AUs;
predict the AUs in a second set of frames of the video; and
determine the facial emotion, by comparing: the registered AUs of the neutral facial expression in the first set of frames of the video, with the predicted AUs in the second set of frames of the video; and

an application unit configured to perform an action based on the determined facial emotion.

7. A computing device for determining a facial emotion, in a video comprising a face with a facial artifact, the computing device comprising:
a processor; and
a memory having instructions stored thereon executed by the processor to perform;
detecting Action Units (AUs) in a first set of frames of the video, wherein the first set of frames of the video comprises the face with the facial artifact, and wherein each AU comprises a fundamental action of an individual muscle or group of muscles;
determining frequently occurring AUs that appear in a larger subset of the first set of frames of the video than other AUs, and registering, as a neutral facial expression, the frequently occurring AUs of the first set of frames of the video; and
predicting the AUs in a second set of frames of the video, and determining the facial emotion, by comparing:

the registered AUs of the neutral facial expression in the first set of frames of the video, with the predicted AUs in the second set of frames of the video.

8. The computing device of claim 7 , wherein the computing device is configured to register the AUs of the neutral facial expression, by assuming that the face in the video shows the neutral facial expression in the first set of frames of the video.

9. The computing device of claim 7 , wherein the computing device is configured to detect the AUs by localizing the face in the video, and extracting features of the localized face.

10. The computing device of claim 7 , wherein a weight of an AU with a facial artifact is reduced.

11. The computing device of claim 7 ,
wherein a non-transitory storage medium is configured to store a trained database that is accessible during real-time estimation of the facial emotion.

12. The computing device of claim 7 , further comprises:
a database of facial images with training data of AUs;
a face detector with a version of an Active Appearance Module;
a feature extractor; and
a Support Vector Machine (SVM) classifier training processor.

13. A computer-implemented method for determining a facial expression, the method comprising:
extracting, as a neutral feature vector, frequently occurring neutral features from a normalized neutral facial image that appear in a larger subset of a sequence of frames of a video than other neutral features, and registering, as a neutral facial expression, the frequently occurring neutral features;
predicting neutral Action Units based on the neutral feature vector, wherein each Action Unit comprises a fundamental action of an individual muscle or group of muscles;
identifying a facial artifact in the predicted neutral Action Units;
assigning a reduced weight to a neutral Action Unit containing the identified facial artifact;
registering, as the neutral facial expression, the reduced weighted neutral Action Unit;
extracting, as a feature vector, features from a normalized facial image;
predicting Action Units based on the feature vector; and
determining the facial expression based on the predicted Action Units and the registered Action Units of the neutral facial expression.

14. A computer-implemented method for recognizing an emotion based on a facial expression, the method comprising:
registering a neutral facial expression, comprising a first set of Action Units that appear in a larger subset of a sequence of frames of a video than other Action Units, and comprising weights based on facial artifacts identified in the neutral facial expression, wherein each Action Unit comprises a fundamental action of an individual muscle or group of muscles;
determining, by a processor, a facial expression based on a second set of Action Units and the registered first set of Action Units for the neutral facial expression; and
recognizing the emotion based on the determined facial expression.

15. The method of claim 14 , wherein a weight of an Action Unit with a facial artifact is reduced, relative to a weight of an Action Unit with no facial artifact.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Balasubramanian Anand
• Sudha Velusamy
• Viswanath Gopalakrishnan
• Anshul Sharma
• Pratibha Moogi

Classifications

• G06K9/00315

  Physics · mapped topic

• G06K9/00281 Primary

  Physics · mapped topic

• G06V40/176 Primary

  Dynamic expression · CPC title

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title