Real-time performance capture with on-the-fly correctives

Patent metadata

Field	Value
Publication number	US-9317954-B2
Application number	US-201314141348-A
Country	US
Kind code	B2
Filing date	Dec 26, 2013
Priority date	Sep 23, 2013
Publication date	Apr 19, 2016
Grant date	Apr 19, 2016

Abstract

Techniques for facial performance capture using an adaptive model are provided herein. For example, a computer-implemented method may include obtaining a three-dimensional scan of a subject and a generating customized digital model including a set of blendshapes using the three-dimensional scan, each of one or more blendshapes of the set of blendshapes representing at least a portion of a characteristic of the subject. The method may further include receiving input data of the subject, the input data including video data and depth data, tracking body deformations of the subject by fitting the input data using one or more of the blendshapes of the set, and fitting a refined linear model onto the input data using one or more adaptive principal component analysis shapes.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining a three-dimensional scan of a subject;
generating a customized digital model including a set of blendshapes using the three-dimensional scan, each of one or more blendshapes of the set of blendshapes representing at least a portion of a characteristic of the subject;
receiving input data of the subject, the input data including video data and depth data;
tracking body deformations of the subject by fitting the input data using one or more of the blendshapes of the set;
generating a refined linear model including one or more adaptive principal component analysis shapes, the refined linear model being generated by deforming the set of blendshapes onto the input data to obtain a deformed mesh and projecting the deformed mesh to a linear adaptive principal component analysis subspace; and
fitting the refined linear model onto the input data using the one or more adaptive principal component analysis shapes.

2. The computer-implemented method of claim 1 , wherein the three-dimensional scan of the subject includes the subject in a neutral position.

3. The computer-implemented method of claim 1 , wherein the video data of the received input data includes markerless video data of the subject performing an act that involves body deformation, the markerless video data including a sequence of frames recorded without using physical markers applied to the subject that are designed for motion capture.

4. The computer-implemented method of claim 1 , wherein the body deformations include deformation of at least part of a face of the subject.

5. The computer-implemented method of claim 1 , wherein no training phase including obtaining a set of pre-processed facial expressions of the subject is required to generate an animation of the subject.

6. The computer-implemented method of claim 1 , wherein the one or more adaptive principal component analysis shapes include one or more anchor shapes and one or more corrective shapes, the one or more anchor shapes being obtained by performing a principal component analysis on the set of blendshapes, and the one or more corrective shapes being based on additional shapes not present in the set of blendshapes.

7. The computer-implemented method of claim 6 , wherein the one or more corrective shapes are iteratively updated by determining new shapes that are outside of the one or more adaptive principal component analysis shapes, and modifying the one or more adaptive principal component analysis shapes based on the determined new shapes.

8. A system, comprising:
a memory storing a plurality of instructions; and
one or more processors configurable to:
obtain a three-dimensional scan of a subject;
generate a customized digital model including a set of blendshapes using the three-dimensional scan, each of one or more blendshapes of the set of blendshapes representing at least a portion of a characteristic of the subject;
receive input data of the subject, the input data including video data and depth data;
track body deformations of the subject by fitting the input data using one or more of the blendshapes of the set;
generate a refined linear model including one or more adaptive principal component analysis shapes, the refined linear model being generated by deforming the set of blendshapes onto the input data to obtain a deformed mesh and projecting the deformed mesh to a linear adaptive principal component analysis subspace; and
fit the refined linear model onto the input data using the one or more adaptive principal component analysis shapes.

9. The system of claim 8 , wherein the three-dimensional scan of the subject includes the subject in a neutral position.

10. The system of claim 8 , wherein the video data of the received input data includes markerless video data of the subject performing an act that involves body deformation, the markerless video data including a sequence of frames recorded without using physical markers applied to the subject that are designed for motion capture.

11. The system of claim 8 , wherein the body deformations include deformation of at least part of a face of the subject.

12. The system of claim 8 , wherein no training phase including obtaining a set of pre-processed facial expressions of the subject is required to generate an animation of the subject.

13. The system of claim 8 , wherein the one or more adaptive principal component analysis shapes include one or more anchor shapes and one or more corrective shapes, the one or more anchor shapes being obtained by performing a principal component analysis on the set of blendshapes, and the one or more corrective shapes being based on additional shapes not present in the set of blendshapes.

14. The system of claim 13 , wherein the one or more corrective shapes are iteratively updated by determining new shapes that are outside of the one or more adaptive principal component analysis shapes, and modifying the one or more adaptive principal component analysis shapes based on the determined new shapes.

15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:
instructions that cause the one or more processors to obtain a three-dimensional scan of a subject;
instructions that cause the one or more processors to generate a customized digital model including a set of blendshapes using the three-dimensional scan, each of one or more blendshapes of the set of blendshapes representing at least a portion of a characteristic of the subject;
instructions that cause the one or more processors to receive input data of the subject, the input data including video data and depth data;
instructions that cause the one or more processors to track body deformations of the subject by fitting the input data using one or more of the blendshapes of the set;
instructions that cause the one or more processors to generate a refined linear model including one or more adaptive principal component analysis shapes, the refined linear model being generated by deforming the set of blendshapes onto the input data to obtain a deformed mesh and projecting the deformed mesh to a linear adaptive principal component analysis subspace; and
instructions that cause the one or more processors to fit the refined linear model onto the input data using the one or more adaptive principal component analysis shapes.

16. The non-transitory computer-readable memory of claim 15 , wherein the three-dimensional scan of the subject includes the subject in a neutral position.

17. The non-transitory computer-readable memory of claim 15 , wherein the video data of the received input data includes markerless video data of the subject performing an act that involves body deformation, the markerless video data including a sequence of frames recorded without using physical markers applied to the subject that are designed for motion capture.

18. The non-transitory computer-readable memory of claim 15 , wherein no training phase including obtaining a set of pre-processed facial expressions of the subject is required to generate an animation of the subject.

19. The non-transitory computer-readable memory of claim 15 , wherein the one or more adaptive principal component analysis shapes include one or more anchor shapes and one or more corrective shapes, the one or more anchor shapes being obtained by performing a principal component analysis on the set of blendshapes, and the one or more corrective shapes being

Assignees

• Lucasfilm Entertainment Co Ltd

Inventors

• Li Hao
• Yu Jihun
• Ye Yuting
• Bregler Christoph

Classifications

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F18/2135

  based on approximation criteria, e.g. principal component analysis · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06K9/00214

  Physics · mapped topic