Automated facial action coding system

Patent metadata

Field	Value
Publication number	US-9530048-B2
Application number	US-201414311814-A
Country	US
Kind code	B2
Filing date	Jun 23, 2014
Priority date	Aug 26, 2008
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

An automatic facial action coding system and method can include processing an image to identify a face in the image, to detect and align one or more facial features shown in the image, and to define one or more windows on the image. One or more distributions of pixels and color intensities can be quantified in each of the one or more windows to derive one or more two-dimensional intensity distributions of one or more colors within the window. The one or more two-dimensional intensity distributions can be processed to select image features appearing in the one or more windows and to classify one or more predefined facial actions on the face in the image. A facial action code score that includes a value indicating a relative amount of the predefined facial action occurring in the face in the image can be determined for the face in the image for each of the one or more predefined facial actions.

First claim

What is claimed:

1. A method comprising:
processing a plurality of image windows located on a face in an image to calculate, for each image window of the plurality, a quantity that is informative of an occurrence of a facial action unit, the quantity comprising a first relative likelihood of the occurrence of the facial action unit, and the processing comprising estimating, for each of the plurality of image windows, the first relative likelihood for the image window, the estimating comprising applying a plurality of image filters to each image window to produce characteristics representing the image window, and processing the characteristics to generate the first relative likelihood for the image window; and
combining the quantities calculated for the plurality of windows to estimate a relative likelihood of the facial action unit occurring in the face;
wherein the processing and the combining are implemented on computer hardware.

2. The method as in claim 1 , wherein the estimating of the plurality of first relative likelihoods comprises use of a machine-learned technique.

3. The method as in claim 1 , wherein the applying of the plurality of image filters to the image window produces image-based representations of the image window.

4. The method as in claim 1 , wherein the processing of the characteristics comprises use of a machine-learned technique.

5. The method as in claim 4 , wherein the machine-learned technique has been trained on other images that include other faces.

6. The method as in claim 4 , wherein the machine-learned technique processes a subset of the characteristics to produce the first relative likelihood for the image window.

7. The method as in claim 1 , wherein the processing of the plurality of image windows further comprises: for each image window of the plurality, also estimating an additional first relative likelihood of an additional facial action unit; and
wherein the method further comprises: also combining the plurality of additional first relative likelihoods to estimate an additional relative likelihood of the additional facial action unit occurring in the face.

8. The method as in claim 7 , wherein the also estimating and the also combining for the additional facial action unit are performed in parallel with the estimating and combining for the facial action unit.

9. The method as in claim 7 , wherein the facial action unit and the additional facial action unit are defined by the Facial Action Coding System (FACS).

10. The method as in claim 7 , further comprising: producing a score vector, the score vector comprising the relative likelihood of the facial action unit and the additional relative likelihood of the additional facial action unit.

11. The method as in claim 7 , further comprising: classifying expressions of emotion based on the relative likelihood of the facial action unit and the additional relative likelihood of the additional facial action unit.

12. The method as in claim 1 , wherein the image comprises an image from a video stream, and wherein the method further comprises:
also processing an additional plurality of image windows located on the face in a second image from the video stream to calculate, for each additional image window of the additional plurality, an additional quantity that is informative of the occurrence of the facial action unit;
also estimating an additional plurality of first relative likelihoods of the facial action unit occurring in each of the additional plurality of image windows; and
also combining the additional plurality of first relative likelihoods to estimate an additional second relative likelihood of the facial action unit occurring in the face.

13. The method as in claim 1 , wherein the plurality of image filters comprises at least one of motion detectors and spatio-temporal filters.

14. The method as in claim 1 , wherein the plurality of image windows comprise image windows of different sizes and located to include different locations on the face.

15. The method as in claim 1 , wherein the plurality of image windows is defined relative to a predefined size and orientation of the face in the image.

16. A non-transitory machine readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
processing a plurality of image windows located on a face in an image to calculate, for each image window of the plurality, a quantity that is informative of an occurrence of a facial action unit, the quantity comprising a first relative likelihood of the occurrence of the facial action unit, and the processing comprising estimating, for each of the plurality of image windows, the first relative likelihood for the image window, the estimating comprising applying a plurality of image filters to each image window to produce characteristics representing the image window, and processing the characteristics to generate the first relative likelihood for the image window; and
combining the quantities calculated for the plurality of windows to estimate a relative likelihood of the facial action unit occurring in the face.

17. A system comprising:
computer hardware configured to perform operations comprising:
processing a plurality of image windows located on a face in an image to calculate, for each image window of the plurality, a quantity that is informative of an occurrence of a facial action unit, the quantity comprising a first relative likelihood of the occurrence of the facial action unit, and the processing comprising estimating, for each of the plurality of image windows, the first relative likelihood for the image window, the estimating comprising applying a plurality of image filters to each image window to produce characteristics representing the image window, and processing the characteristics to generate the first relative likelihood for the image window; and
combining the quantities calculated for the plurality of windows to estimate a relative likelihood of the facial action unit occurring in the face.

Assignees

• Univ California
• The Res Found Of State Univ Of New York

Inventors

• Bartlett Marian Stewart
• Littlewort-Ford Gwen
• Movellan Javier
• Fasel Ian
• Frank Mark

Classifications

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V10/56 Primary

  relating to colour · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06K9/00281

  Physics · mapped topic

• G06K9/4652

  Physics · mapped topic