Identifying facial expressions in acquired digital images

Patent metadata

Field	Value
Publication number	US-9818024-B2
Application number	US-201514680977-A
Country	US
Kind code	B2
Filing date	Apr 7, 2015
Priority date	May 20, 2009
Publication date	Nov 14, 2017
Grant date	Nov 14, 2017

Abstract

A face is detected and identified within an acquired digital image. One or more features of the face is/are extracted from the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both. A model including multiple shape parameters is applied to the two independent eyes or subsets of features of each of the two eyes, and/or to the lips or partial lips or one or more other mouth features and one or both eyes. One or more similarities between the one or more features of the face and a library of reference feature sets is/are determined. A probable facial expression is identified based on the determining of the one or more similarities.

First claim

What is claimed is:

1. A method for mirroring a user facial expression in a three-dimensional (3D) in-game avatar in real-time, the method comprising:
generating a 3D in-game avatar and displaying the 3D in-game avatar within a game display of a game by performing:
capturing an image of a user as the user participates in the game;
based on contents of the image, determining a plurality of user facial features in the image depicting a user facial expression of the user, wherein the plurality of user facial features comprises a left eye region, a right eye region, and a lips region;
applying an active appearance face model, which includes a plurality of facial feature parameters, to the left eye region, the right eye region and the lips region to determine similarities between the left eye region and a reference left eye feature defined by one or more facial feature parameters of the plurality of facial feature parameters, similarities between the right eye region and a reference right eye feature defined by one or more facial feature parameters of the plurality of facial feature parameters, and similarities between the lips region and a reference lips feature defined by one or more facial feature parameters of the plurality of facial feature parameters;
generating a left eye sub-model based on the similarities between the left eye region and the reference left eye feature;
generating a right eye sub-model based on the similarities between the right eye region and the reference right eye feature;
generating a lips sub-model based on the similarities between the lips region and the reference lips feature;
generating a global model by projecting and fitting the left eye sub-model onto the left eye region, projecting and fitting the right eye sub-model onto the right eye region, and projecting and fitting the lips sub-model onto the lips region; and
based on the global model, generating a 3D face model of the 3D in-game avatar; and
causing the 3D in-game avatar to mimic the user facial expression in the game display in real-time.

2. The method of claim 1 ,
wherein the determining of the user facial expression is performed based on one or more features depicted in the image;
wherein the one or more features comprise one or more of: geometric shapes, deformations of facial components, pose variations, appearance features;
wherein the appearance features comprise one or more skin texture characteristics; and
wherein the one or more skin texture characteristics comprise one or more of: furrows, bulges, expression wrinkles, illumination variations, blushing.

3. The method of claim 1 , wherein the determining of the user facial expression is based on, at least in part, one or more of: multiple pose variations, multiple directional illumination conditions.

4. The method of claim 1 , wherein the determining of the user facial expression comprises:
detecting a face in the image;
determining one or more parameters specific to the detected face;
determining one or more values for the one or more parameters specific to the detected face; and
based on, at least in part, the one or more values, determining the user facial expression.

5. The method of claim 1 , further comprising categorizing the user facial expression as indicating one or more of: surprise, fear, happiness, anger, neutral, sadness, disgust.

6. The method of claim 1 , further comprising adapting a difficulty of the game based on the user facial expression.

7. The method of claim 1 , further comprising adapting a game workflow of the game based on the user facial expression.

8. A non-transitory computer-readable storage medium, storing one or more computer instructions which, when executed by one or more processors, cause the processors to perform:
generating a 3D in-game avatar and displaying the 3D in-game avatar within a game display of a game by performing:
capturing an image of a user as the user participates in the game;
based on contents of the image, determining a plurality of user facial features in the image depicting a user facial expression of the user, wherein the plurality of user facial features comprises a left eye region, a right eye region and a lips region;
applying an active appearance face model, which includes a plurality of facial feature parameters, to the left eye region, the right eye region and the lips region to determine similarities between the left eye region and a reference left eye feature defined by one or more facial feature parameters of the plurality of facial feature parameters, similarities between the right eye region and a reference right eye feature defined by one or more facial feature parameters of the plurality of facial feature parameters, and similarities between the lips region and a reference lips feature defined by one or more facial feature parameters of the plurality of facial feature parameters;
generating a left eye sub-model based on the similarities between the left eye region and the reference left eye feature;
generating a right eye sub-model based on the similarities between the right eye region and the reference right eye feature;
generating a lips sub-model based on the similarities between the lips region and the reference lips feature;
generating a global model by projecting and fitting the left eye sub-model onto the left eye region, projecting and fitting the right eye sub-model onto the right eye region, and projecting and fitting the lips sub-model onto the lips region; and
based on the global model, generating a 3D face model of the 3D in-game avatar and
causing the 3D in-game avatar to mimic the user facial expression in the game display in real-time.

9. The non-transitory computer-readable storage medium of claim 8 ,
wherein the determining of the user facial expression is performed based on one or more features depicted in the image;
wherein the one or more features comprise one or more of: geometric shapes, deformations of facial components, pose variations, appearance features;
wherein the appearance features comprise one or more skin texture characteristics; and
wherein the one or more skin texture characteristics comprise one or more of: furrows, bulges, expression wrinkles, illumination variations, blushing.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the determining of the user facial expression is based on, at least in part, one or more of: multiple pose variations, multiple directional illumination conditions.

11. The non-transitory computer-readable storage medium of claim 8 , further comprising instructions for:
detecting a face in the image;
determining one or more parameters specific to the detected face;
determining one or more values for the one or more parameters specific to the detected face; and
based on, at least in part, the one or more values, determining the user facial expression.

12. The non-transitory computer-readable storage medium of claim 8 , further comprising instructions for categorizing the user facial expression as indicating one or more of: surprise, fear, happiness, anger, neutral, sadness, disgust.

13. The non-transitory computer-readable storage medium of claim 8 , further comprising instructions for adapting a difficulty of the game based on the user facial expression.

14. The non-transitory computer-readable storage medium of claim 8 , further comprising instructions for adapting a game workflow of the game based on the user facial expression.

15. An apparatus for mirroring a user facial expression in a three-dimensional (3D) in-game avatar in real-time, the apparatus comprising:
a us

Assignees

• Fotonation Ltd

Inventors

• Bacivarov Ioana
• Corcoran Peter

Classifications

• G06V40/174

  Facial expression recognition · CPC title

• G06V40/175 Primary

  Static expression · CPC title

• H04N23/611

  where the recognised objects include parts of the human body · CPC title

• G06K9/00302 Primary

  Physics · mapped topic

• G06K9/00281

  Physics · mapped topic