Lip synchronization between rigs

Patent metadata

Field	Value
Publication number	US-10055874-B2
Application number	US-201514831021-A
Country	US
Kind code	B2
Filing date	Aug 20, 2015
Priority date	Aug 20, 2015
Publication date	Aug 21, 2018
Grant date	Aug 21, 2018

Abstract

In some embodiments a method of transferring facial expressions from a subject to a computer-generated character is provided where the method includes receiving positional information from a motion capture session of the subject representing a performance having facial expressions to be transferred to the computer-generated character, receiving a first animation model that represents the subject, and receiving a second animation model that represents the computer-generated character. Each of the first and second animation models can include a plurality of adjustable controls that define geometries of the model and that can be adjusted to present different facial expressions on the model, and where the first and second animation models are designed so that setting the same values for the same set of adjustable controls in each model generates a similar facial poses on the models. The method further includes determining a solution, including values for at least some of the plurality of controls, that matches the first animation model to the positional information to reproduce the facial expressions from the performance to the first animation model, retargeting the facial expressions from the performance to the second animation model using the solution; and thereafter, synchronizing lip movement of the second animation model with lip movement from the first animation model.

First claim

What is claimed is:

1. A method of transferring facial expressions from a subject to a computer-generated character, the method comprising:
receiving positional information from a motion capture session of the subject representing a performance having facial expressions to be transferred to the computer-generated character;
receiving a first animation model that represents the subject and a second animation model that represents the computer-generated character, each of the first and second animation models including a plurality of adjustable controls that define geometries of the model and that can be adjusted to present different facial expressions on the model, wherein setting the same values for corresponding sets of adjustable controls in the models generates a similar facial poses on the models;
determining a solution that matches the first animation model to the positional information to reproduce the facial expressions from the performance to the first animation model, the solution including values for at least some of the plurality of adjustable controls;
retargeting the facial expressions from the performance to the second animation model using the solution; and
thereafter, synchronizing lip movement of the second animation model with lip movement from the first animation model.

2. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 1 wherein synchronizing lip movement of the second animation model with lip movement from the first animation model comprises:
determining a visible percentage of teeth for the first animation model;
determining a visible percentage of teeth for the second animation model; and
determining if the visible percentage of teeth for the second animation model matches the visible percentage of teeth for the first animation model, and if not, adjusting the visible percentage of teeth in the second animation model to more closely match the percentage in the first animation model.

3. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 2 wherein adjusting the visible percentage of teeth in the second animation model to more closely match the percentage in the first animation model includes moving either or both upper and lower lip shapes of the second animation model.

4. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 3 wherein a match between the visible percentage of teeth for the second animation model and the visible percentage of teeth for the first animation model occurs when the percentages are within a predetermined amount of each other.

5. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 3 wherein the lip shapes are moved according to a binary search algorithm, and wherein the method further comprises repeating a sequence of: (i) determining a visible percentage of teeth for the second animation model, and (ii) determining if the visible percentage of teeth for the second animation model matches the visible percentage of teeth for the first animation model, and if not, adjusting the visible percentage of teeth in the second animation model to more closely match the percentage in the first animation model until the visible percentages match.

6. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 3 wherein:
determining a visible percentage of teeth for the first animation model includes: (i) determining an occluding contour of both the upper or lower lips of the first animation model, (ii) determining a distance between a root and a tip for each of the upper and lower teeth of the first animation model, (iii) computing an intersection between the occluding contour and the tooth lines of the upper and lower teeth of the first animation model, and (iv) calculating a percentage of visible upper teeth and a percentage of visible lower teeth for the first animation model; and
determining a visible percentage of teeth for the second animation model includes: (i) determining an occluding contour of both the upper or lower lips of the second animation model, (ii) determining a distance between a root and a tip for each of the upper and lower teeth of the second animation model, (iii) computing an intersection between the occluding contour and the tooth lines of the upper and lower teeth of the second animation model, and (iv) calculating a percentage of visible upper teeth and a percentage of visible lower teeth for the second animation model.

7. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 1 wherein synchronizing lip movement of the creature with lip movement from the first animation model comprises:
determining a jaw opening of the first animation model;
determining a jaw opening for the second animation model; and
determining if the jaw opening of the second animation model matches the jaw opening of the first animation model, and if not, adjusting the jaw opening of the second animation model to more closely match the jaw opening of the first animation model.

8. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 7 wherein determining if the jaw opening of the second animation model matches the jaw opening of the first animation model is performed only if the jaw opening of the second animation model is less than a predetermined amount.

9. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 7 wherein determining a jaw opening for each of the actor and second animation models includes:
determining a distance between roots of upper and lower teeth of the respective rig;
determining a distance between tips of upper and lower teeth of the respective rig; and
computing a ratio of the distance between the roots of the upper and lower teeth and the distance between the tips of the upper and lower teeth.

10. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 7 wherein adjusting the jaw opening of the second animation model to more closely match the jaw opening of the first animation model includes raising or lowering the lower jaw.

11. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 7 wherein the jaw opening is adjusted according to a binary search algorithm and wherein the method further comprises repeating a sequence of: (i) determining a jaw opening of the second animation model, and (ii) determining if the jaw opening of the second animation model matches the jaw opening of the first animation model, and if not, adjusting the jaw opening of the second animation model to more closely match the jaw opening of the first animation model until the jaw openings match.

12. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 7 wherein a match between the jaw opening of the second animation model and the jaw opening of the first animation model occurs when the openings are within a predetermined amount of each other.

13. The method of transferring facial expressions from a subject to a computer-generated character set forth in claim 1 wherein synchronizing lip movement of the creature with lip movement from the first animation model comprises:
determining a jaw opening of the first animation model, determining a jaw opening for the s

Assignees

• Lucasfilm Entertainment Co Ltd

Inventors

• Yu Ji Hun
• Koperwas Michael
• Yost Jeffrey Bruce
• Santos Sheila
• Bhat Kiran S

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T2207/30204

  Marker · CPC title

• G06V40/174

  Facial expression recognition · CPC title