Determining control values of an animation model using performance capture

Patent metadata

Field	Value
Publication number	US-10147219-B2
Application number	US-201715424727-A
Country	US
Kind code	B2
Filing date	Feb 3, 2017
Priority date	May 5, 2015
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

Performance capture systems and techniques are provided for capturing a performance of a subject and reproducing an animated performance that tracks the subject's performance. For example, systems and techniques are provided for determining control values for controlling an animation model to define features of a computer-generated representation of a subject based on the performance. A method may include obtaining input data corresponding to a pose performed by the subject, the input data including position information defining positions on a face of the subject. The method may further include obtaining an animation model for the subject that includes adjustable controls that control the animation model to define facial features of the computer-generated representation of the face, and matching one or more of the positions on the face with one or more corresponding positions on the animation model. The matching includes using an objective function to project an error onto a control space of the animation model. The method may further include determining, using the projected error and one or more constraints on the adjustable controls, one or more values for one or more of the adjustable controls. The values are configured to control the animation model to cause the computer-generated representation to perform a representation of the pose using the one or more adjustable controls.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining input data corresponding to a pose of a subject, the input data including position information for points on a face of the subject;
obtaining an animation model, the animation model being associated with adjustable controls usable for adjusting the animation model to define poses for a computer representation of a face;
determining, using the position information, a correspondence between one or more of the points on the face of the subject and one or more points of the computer representation of the face, wherein the determining includes using an objective function to project an error onto a control space of the animation model; and
setting the one or more adjustable controls to have one or more values based on the projected error and one or more constraints for the one or more adjustable controls, wherein the setting of the one or more adjustable controls causes the computer representation of the face to have a pose based on the pose of the subject.

2. The method of claim 1 , wherein determining the correspondence between the one or more of the points on the face of the subject and the one or more points of the computer representation of the face includes:
determining a vector between a point on the face and a corresponding point of the animation model, wherein the vector represents the error; and
projecting the vector representing the error onto the control space of the animation model.

3. The method of claim 2 , wherein projecting the vector onto the control space of the animation model includes:
determining a first error projection, including determining a first dot product of the determined vector and the first vector along a direction of the first vector, the first vector representing a first adjustable control of the animation model;
determining a second error projection, including determining a second dot product of the determined vector, minus the first error projection, and a second vector along a direction of the second vector, the second vector representing a second adjustable control of the animation model; and
adding the first error projection and the second error projection.

4. The method of claim 1 , wherein the one or more constraints include one or more of a complimentary constraint that prevents two or more of the adjustable controls from being simultaneously used for controlling a facial feature, a limit constraint that limits an amount by which an adjustable control can be adjusted to a range of values, a velocity constraint that limits an amount by which an adjustable control can change between successive frames, and a symmetric constraint that combines two or more controls.

5. The method of claim 1 , further comprising:
determining the one or more values for the one or more of the adjustable controls using an animation guide that is specific to the subject, wherein the animation guide limits the adjustable controls that can be used to control the facial features for the subject.

6. The method of claim 1 , wherein the input data further includes contour information defining one or more contours of the face, and further comprising:
minimizing a distance between at least one of the one or more contours of the face and data representing one or more corresponding contours of the animation model.

7. The method of claim 6 , wherein the contour information defining the one or more contours of the face includes two-dimensional data.

8. The method of claim 6 , wherein the one or more contours of the face include an inner eye contour of the subject or an inner mouth contour of the subject.

9. The method of claim 6 , wherein minimizing the distance between the at least one contour of the face and the one or more corresponding contours of the animation model includes matching edges of the at least one contour of the face with the one or more corresponding contours of the animation model.

10. The method of claim 1 , wherein the points on the face are determined from markers applied to the face.

11. The method of claim 1 , wherein the input data is determined from video data of the face of the subject, and wherein the video data is received from one or more head-mounted cameras.

12. The method of claim 11 , wherein the position information for the points on the face includes two-dimensional data and three-dimensional data, the two-dimensional data corresponding to points on the face that are visible from one of the one or more head-mounted cameras, and the three-dimensional data corresponding to points on the face that are visible from two of the one or more head-mounted cameras.

13. The method of claim 1 , wherein the animation model includes one or more blendshapes that are configured to change the facial features of the computer-generated representation of the face, and wherein the adjustable controls are configured to control the blendshapes.

14. The method of claim 1 , further comprising outputting the values for the one or more adjustable controls for controlling an animated representation of a character using a character animation model, the character animation model including adjustable controls that control the character animation model to define facial features of the animated representation of the character, wherein the outputted one or more values are configured to control the character animation model to cause the animated representation of the character to have a pose based on the pose of the subject.

15. A system, comprising:
a memory storing a plurality of instructions; and
one or more processors configurable to:
obtain input data corresponding to a pose of a subject, the input data including position information for points on a face of the subject;
obtain an animation model, the animation model being associated with adjustable controls usable for adjusting the animation model to define poses for a computer representation of a face;
determine, using the position information, a correspondence between one or more of the points on the face of the subject and one or more points of the computer representation of the face, wherein the determining includes using an objective function to project an error onto a control space of the animation model; and
set the one or more adjustable controls to have one or more values based on the projected error and one or more constraints for the one or more adjustable controls, wherein the setting of the one or more adjustable controls causes the computer representation of the face to have a pose based on the pose of the subject.

16. The system of claim 15 , wherein determining the correspondence between the one or more of the points on the face of the subject and the one or more points of the computer representation of the face includes:
determining a vector between a position on the face and a corresponding position on the animation model, wherein the vector represents the error; and
projecting the vector representing the error onto the control space of the animation model.

17. The system of claim 15 , wherein the one or more constraints include one or more of a complimentary constraint that prevents two or more of the adjustable controls from being simultaneously used for controlling a facial feature, a limit constraint that limits an amount by which an adjustable control can be adjusted to a range of values, a velocity constraint that limits an amount by which an adjustable control can change between successive frames, and a symmetric constraint that combines two or more controls.

18. A non-transitory computer-

Assignees

• Lucasfilm Entertainment Co Ltd

Inventors

• Bhat Kiran
• Koperwas Michael
• Yost Jeffery
• Yu Ji Hun
• Santos Sheila

Classifications

• G06T7/75 Primary

  involving models · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06K9/00315

  Physics · mapped topic