Avatar facial expression representation in multidimensional space

Patent metadata

Field	Value
Publication number	US-10789753-B2
Application number	US-201916359851-A
Country	US
Kind code	B2
Filing date	Mar 20, 2019
Priority date	Apr 23, 2018
Publication date	Sep 29, 2020
Grant date	Sep 29, 2020

Abstract

Examples of the disclosed systems and methods may provide for improved and more realistic rendering of virtual characters and a more realistic interaction between a user and virtual characters. For example, the systems and methods describe techniques for mathematically generating a map used for animating facial expressions in a multidimensional animation blendspace. As another example, the systems and methods describe a transition system for dynamically transitioning facial expressions across a face of the virtual character. As another example, realistic physical movements can be added to a virtual character's facial expressions to provide interactivity with other virtual characters.

First claim

What is claimed is:

1. A system for generating a map of facial expressions, the system comprising:
non-transitory storage medium storing vector values for facial expressions; and
a hardware processor programmed to:
identify a set of facial expressions and a neutral expression to be projected onto a map, wherein each facial expression is represented by an expression specific vector and the neutral expression is represented by a neutral vector;
calculate a first expression specific delta vector of a first facial expression based on a first expression specific vector of the first facial expression and the neutral vector;
calculate a first length of the first expression specific delta vector;
calculate a second expression specific delta vector of a second facial expression based on a second expression specific vector of the second facial expression and the neutral vector;
calculate a second length of the second expression specific delta vector;
calculate an angular relationship between the first expression specific delta vector and the second expression specific delta vector; and
project the first facial expression and the second facial expression onto a map,
wherein locations of the first facial expression and second facial expression, respectively, with reference to the neutral expression are determined based at least in part on the first length and the second length, and
wherein a relative position between the first facial expression and the second facial expression is determined based at least in part on the angular relationship between the first expression specific delta vector and the second expression specific delta vector.

2. The system of claim 1 , wherein the map comprises two-dimensional projections of the set of facial expressions.

3. The system of claim 1 , wherein the map comprises a plurality of arms with each arm being positioned with one or more facial expressions, and each arm of the plurality of arms is connected to the neutral expression.

4. The system of claim 3 , wherein the map is wheel-shaped, such that the neutral expression is placed at the center of the wheel-shaped map.

5. The system of claim 4 , wherein two facial expressions on an arm have two expression specific delta vectors whose angular relationship corresponds to an angle less than 10 degrees.

6. The system of claim 5 , wherein the two expression specific delta vectors correspond to two similar facial expressions with different intensities.

7. The system of claim 1 , wherein the expression specific vector comprises a plurality of dimensions, where each dimension is usable to drive deformations of a region of a mesh of a virtual character's face.

8. The system of claim 7 , wherein the expression specific vector comprises over 100 dimensions.

9. The system of claim 1 , wherein the hardware processor is programmed to:
determine a position of an animation blend cursor;
identify a plurality of facial expressions around the animation blend cursor wherein the plurality of facial expressions form a polygon;
determine weights corresponding to the plurality of facial expressions based at least in part on the position of the animation blend cursor with respective locations of the plurality of facial expressions on the map;
generate an animation of a facial expression outside of the set of facial expressions projected onto the map based on the plurality of facial expressions and the weights corresponding to the plurality of facial expressions.

10. The system of claim 1 , wherein the hardware processor is further programmed to:
calculate additional facial expression delta vectors of respective additional facial expressions; and
calculate angular relationships between the first expression specific delta vector and the additional facial expression delta vectors;
wherein relative positions between the first facial expression and the additional facial expressions are determined based at least in part on respective calculated angular relationships.

11. The system of claim 1 , wherein:
to calculate a first expression specific delta vector, the hardware processor is programmed to subtract the neutral vector from the first expression specific vector.

12. The system of claim 1 , wherein to calculate the angular relationship between the first expression specific delta vector and the second expression specific delta vector, the hardware processor is programmed to calculate a dot product between the first expression specific delta vector and the second expression specific delta vector.

13. A method for generating a map of facial expressions, the method comprising:
identifying a set of facial expressions to be projected onto a map, wherein each facial expression is represented by an expression specific vector;
taking a difference between a first expression specific vector of a first facial expression and a reference vector to generate a first expression specific delta vector;
taking another difference between a second expression specific vector of a second facial expression and the reference vector to generate a second expression specific delta vector;
determining a dot product between the first expression specific delta vector and the second expression specific delta vector; and
project the first facial expression and the second facial expression onto a map, wherein a relative position between the first facial expression and the second facial expression is determined based at least in part on the dot product between the first expression specific delta vector and the second expression specific delta vector.

14. A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:
identify a set of facial expressions and a neutral expression to be projected onto a map, wherein each facial expression is represented by an expression specific vector and the neutral expression is represented by a neutral vector;
calculate a first expression specific delta vector of a first facial expression based on a first expression specific vector of the first facial expression and the neutral vector;
calculate a first length of the first expression specific delta vector;
calculate a second expression specific delta vector of a second facial expression based on a second expression specific vector of the second facial expression and the neutral vector;
calculate a second length of the second expression specific delta vector;
calculate an angular relationship between the first expression specific delta vector and the second expression specific delta vector; and
project the first facial expression and the second facial expression onto a map,
wherein locations of the first facial expression and second facial expression, respectively, with reference to the neutral expression are determined based at least in part on the first length and the second length, and
wherein a relative position between the first facial expression and the second facial expression is determined based at least in part on the angular relationship between the first expression specific delta vector and the second expression specific delta vector.

15. The non-transitory computer readable medium of claim 14 , wherein the map comprises two-dimensional projections of the set of facial expressions.

16. The non-transitory computer readable medium of claim 14 , wherein the map comprises a plurality of arms with each arm being positioned with one or more facial expressions, and each arm of the plurality of arms is c

Assignees

• Magic Leap Inc

Inventors

• Miller Iv Thomas Marshall
• Quach The Hung
• Lin Jeffrey

Classifications

• G06V40/174

  Facial expression recognition · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title