Online modeling for real-time facial animation

Patent metadata

Field	Value
Publication number	US-10586372-B2
Application number	US-201916259818-A
Country	US
Kind code	B2
Filing date	Jan 28, 2019
Priority date	Jun 7, 2013
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

Embodiments relate to a method for real-time facial animation, and a processing device for real-time facial animation. The method includes providing a dynamic expression model, receiving tracking data corresponding to a facial expression of a user, estimating tracking parameters based on the dynamic expression model and the tracking data, and refining the dynamic expression model based on the tracking data and estimated tracking parameters. The method may further include generating a graphical representation corresponding to the facial expression of the user based on the tracking parameters. Embodiments pertain to a real-time facial animation system.

First claim

The invention claimed is:

1. A method for real-time facial animation, comprising:
providing a dynamic expression model that includes a plurality of blendshapes;
receiving tracking data from a plurality of frames in a temporal sequence, the tracking data corresponding to facial expressions of a user;
estimating one or more weight values of the blendshape for each of the plurality of frames, wherein the weight values are based on a combined 2D/3D registration of the dynamic expression model; and
generating a graphical representation of a particular facial expression of the user based on the estimated weight values of the blendshapes for the plurality of frames.

2. The method of claim 1 , further comprising:
rendering an animated visual representation of the user based on the graphical representation.

3. The method of claim 1 further comprising:
receiving further tracking data corresponding to facial expressions of the user;
estimating updated weight values based on the further tracking data; and
refining the dynamic expression model based on the estimated updated weight values.

4. The method of claim 1 , further comprising:
receiving neutral tracking data corresponding to a neutral facial expression of the user; and
initializing the dynamic expression model using the neutral tracking data.

5. The method of claim 1 , wherein the plurality of blendshapes each correspond to a unique facial expression.

6. The method of claim 1 , wherein the dynamic expression model further includes a template blendshape model, and wherein the method further comprises:
approximating one or more of the plurality of blendshapes based on the template blendshape model.

7. The method of claim 1 , wherein estimating weight values is repeated based on each of the plurality of frames in the temporal sequence.

8. A non-transitory computer readable medium comprising computer readable code for real-time facial animation, executable by one or more processors to:
provide a dynamic expression model that includes a plurality of blendshapes;
receive tracking data from a plurality of frames in a temporal sequence, the tracking data corresponding to facial expressions of a user;
estimate one or more weight values of the blendshape for each of the plurality of frames, wherein the weight values are based on a combined 2D/3D registration of the dynamic expression model; and
generate a graphical representation of a particular facial expression of the user based on the estimated weight values of the blendshapes for the plurality of frames.

9. The non-transitory computer readable medium of claim 8 , further comprising computer readable code to:
render an animated visual representation of the user based on the graphical representation.

10. The non-transitory computer readable medium of claim 8 , further comprising computer readable code to:
receive further tracking data corresponding to facial expressions of the user;
estimate updated weight values based on the further tracking data; and
refine the dynamic expression model based on the estimated updated weight values.

11. The non-transitory computer readable medium of claim 8 , further comprising computer readable code to:
receive neutral tracking data corresponding to a neutral facial expression of the user; and
initialize the dynamic expression model using the neutral tracking data.

12. The non-transitory computer readable medium of claim 8 , wherein the plurality of blendshapes each correspond to a unique facial expression.

13. The non-transitory computer readable medium of claim 8 , wherein the dynamic expression model further includes a template blendshape model, and wherein the computer readable code further comprises computer readable code to:
approximate one or more of the plurality of blendshapes based on the template blendshape model.

14. The non-transitory computer readable medium of claim 8 , wherein the computer readable code to estimate weight values is repeated based on each of the plurality of frames in the temporal sequence.

15. A system for real-time facial animation, comprising:
one or more processors; and
one or more storage devices coupled to the one or more processors, and comprising computer readable code executable by the one or more processors to:
provide a dynamic expression model that includes a plurality of blendshapes;
receive tracking data from a plurality of frames in a temporal sequence, the tracking data corresponding to facial expressions of a user;
estimate one or more weight values of the blendshape for each of the plurality of frames, wherein the weight values are based on a combined 2D/3D registration of the dynamic expression model; and
generate a graphical representation of a particular facial expression of the user based on the estimated weight values of the blendshapes for the plurality of frames.

16. The system of claim 15 , further comprising computer readable code to:
render an animated visual representation of the user based on the graphical representation.

17. The system of claim 15 , further comprising computer readable code to:
receive further tracking data corresponding to facial expressions of the user;
estimate updated weight values based on the further tracking data; and
refine the dynamic expression model based on the estimated updated weight values.

18. The system of claim 15 , further comprising computer readable code to:
receive neutral tracking data corresponding to a neutral facial expression of the user; and
initialize the dynamic expression model using the neutral tracking data.

19. The system of claim 15 , wherein the plurality of blendshapes each correspond to a unique facial expression.

20. The system of claim 15 , wherein the dynamic expression model further includes a template blendshape model, and wherein the computer readable code further comprises computer readable code to:
approximate one or more of the plurality of blendshapes based on the template blendshape model.

Assignees

• Apple Inc

Inventors

• Bouaziz Sofien
• Pauly Mark

Classifications

• G06T7/292

  Multi-camera tracking · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06K9/00315

  Physics · mapped topic