Online modeling for real-time facial animation

Patent metadata

Field	Value
Publication number	US-11348299-B2
Application number	US-202016773133-A
Country	US
Kind code	B2
Filing date	Jan 27, 2020
Priority date	Jun 7, 2013
Publication date	May 31, 2022
Grant date	May 31, 2022

Abstract

Embodiments relate to a method for real-time facial animation, and a processing device for real-time facial animation. The method includes providing a dynamic expression model, receiving tracking data corresponding to a facial expression of a user, estimating tracking parameters based on the dynamic expression model and the tracking data, and refining the dynamic expression model based on the tracking data and estimated tracking parameters. The method may further include generating a graphical representation corresponding to the facial expression of the user based on the tracking parameters. Embodiments pertain to a real-time facial animation system.

First claim

The invention claimed is:

1. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
obtain, from a first device, a user-generic dynamic expression model comprising a plurality of user-generic blendshapes;
receive, by a local device, tracking data corresponding to one or more facial expressions of a user;
estimate a plurality of weights for the plurality of user-generic blendshapes based on the tracking data;
generate a first graphical representation of the user based on the user-generic dynamic expression model from the local device and the plurality of weights;
generate, by the local device, a user-specific dynamic expression model by refining the user-generic dynamic expression model based on the tracking data and the plurality of weights, wherein the user-specific dynamic expression model is specific to a facial geometry of the user; and
store, by the local device, the user-specific dynamic expression model as associated with the user,
wherein, upon receiving additional tracking data, the local device generates a second graphical representation of the user using the user-specific dynamic expression model and a plurality of weights for user-specific blendshapes based on the additional tracking data.

2. The non-transitory computer readable medium of claim 1 , wherein the tracking data is received from a video frame capturing the one or more facial expressions of the user.

3. The non-transitory computer readable medium of claim 1 , further comprising computer readable code to generate, based on the refined dynamic expression model, a graphic representation corresponding to the one or more facial expressions of the user.

4. The non-transitory computer readable medium of claim 1 , wherein the user-generic blendshapes comprise a neutral model and one or more expressive models.

5. The non-transitory computer readable medium of claim 1 , wherein each of the plurality of user-generic blendshapes comprises a user-generic identity model.

6. The non-transitory computer readable medium of claim 5 , wherein refining the user-generic dynamic expression model comprises applying a corrective deformation field to at least one of the plurality of user-generic blendshapes.

7. The non-transitory computer readable medium of claim 5 , wherein the user-generic identity model comprises a user-generic 3D mesh.

8. A system for generating a dynamic expression model comprising:
one or more processors; and
one or more non-transitory computer readable media comprising computer readable code executable by the one or more processors to:
obtain, from a first device, a user-generic dynamic expression model comprising a plurality of user-generic blendshapes;
receive, by a local device, tracking data corresponding to one or more facial expressions of a user;
estimate a plurality of weights for the plurality of user-generic blendshapes based on the tracking data;

generate a first graphical representation of the user based on the user-generic dynamic expression model from the local device and the plurality of weights;
generate, by the local device, a user-specific dynamic expression model by refining the user-generic dynamic expression model based on the tracking data and plurality of weights, wherein the user-specific dynamic expression model is specific to a facial geometry of the user; and
store, by the local device, the user-specific dynamic expression model as associated with the user,
wherein, upon receiving additional tracking data, the local device generates a second graphical representation of the user using the user-specific dynamic expression model and a plurality of weights for user-specific blendshapes based on the additional tracking data.

9. The system of claim 8 , wherein the tracking data is received from a video frame capturing the one or more facial expressions of the user.

10. The system of claim 8 , further comprising computer readable code to generate, based on the refined dynamic expression model, a graphic representation corresponding to the one or more facial expressions of the user.

11. The system of claim 8 , wherein the user-generic blendshapes comprise a neutral model and one or more expressive models.

12. The system of claim 8 , wherein each of the plurality of user-generic blendshapes comprises a user-generic identity model.

13. The system of claim 12 , wherein refining the user-generic dynamic expression model comprises applying a corrective deformation field to at least one of the plurality of user-generic blendshapes.

14. The system of claim 12 , wherein the user-generic identity model comprises a user-generic 3D mesh.

15. A method for generating a dynamic expression model, comprising:
obtaining, from a first device, a user-generic dynamic expression model comprising a plurality of user-generic blendshapes;
receiving, by a local device, tracking data corresponding to one or more facial expressions of a user;
estimating a plurality of weights for the plurality of user-generic blendshapes based on the tracking data;
generating a first graphical representation of the user based on the user-generic dynamic expression model from the local device and the plurality of weights;
generating, by the local device, a user-specific dynamic expression model by refining the user-generic dynamic expression model based on the tracking data and the plurality of weights, wherein the user-specific dynamic expression model is specific to a facial geometry of the user; and
storing, by the local device, the user-specific dynamic expression model as associated with the user,
wherein, upon receiving additional tracking data, the local device generates a second graphical representation of the user using the user-specific dynamic expression model and a plurality of weights for user-specific blendshapes based on the additional tracking data.

16. The method of claim 15 , wherein the tracking data is received from a video frame capturing the one or more facial expressions of the user.

17. The method of claim 15 , further comprising generating, based on the refined dynamic expression model, a graphic representation corresponding to the one or more facial expressions of the user.

18. The method of claim 15 , wherein each of the plurality of user-generic blendshapes comprises a user-generic identity model.

19. The method of claim 18 , wherein refining the user-generic dynamic expression model comprises applying a corrective deformation field to at least one of the plurality of user-generic blendshapes.

20. The method of claim 18 , wherein the user-generic identity model comprises a user-generic 3D mesh.

Assignees

• Apple Inc

Inventors

• Bouaziz Sofien
• Pauly Mark

Classifications

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06V40/176

  Dynamic expression · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title

• G06T7/292

  Multi-camera tracking · CPC title