Geometry tracking

Patent metadata

Field	Value
Publication number	US-9792479-B2
Application number	US-201414167838-A
Country	US
Kind code	B2
Filing date	Jan 29, 2014
Priority date	Nov 29, 2011
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

A method of motion capture may include accessing a 3D model of a subject, and associating the 3D model of the subject with a 2D representation of the subject in a plurality of frames. The method may also include identifying a change to the 2D representation of the subject between two or more of the plurality of frames, and deforming the 3D model in a virtual 3D space. In some embodiments, the deforming may be based on the identified change to the 2D representation and at least one constraint restricting how the 3D model can be deformed.

First claim

What is claimed is:

1. A computer implemented method of motion capture, the method comprising:
accessing a 3D model;
associating the 3D model with a 2D representation of a subject in a plurality of frames, wherein the plurality of frames captures the subject in a real 3D space;
identifying one or more motion vectors derived from a change to the position of the 2D representation of the subject between two or more of the plurality of frames; and
deforming the 3D model in a virtual 3D space, wherein the deforming is based on:
the identified change to the position of the 2D representation; and
at least one constraint restricting how the 3D model can be deformed,

wherein the at least one constraint comprises the one or more motion vectors.

2. The computer implemented method of motion capture of claim 1 , wherein each frame in the plurality of frames comprises a 2D image of the subject in a real 3D space.

3. The computer implemented method of motion capture of claim 1 , further comprising projecting the identified change onto the virtual 3D space as 3D constraints.

4. The computer implemented method of motion capture of claim 1 , wherein deforming the 3D model in the virtual 3D space is performed in a single use of a non-linear solver.

5. The computer implemented method of motion capture of claim 1 , identifying a change to the 2D representation of the subject is based on texture tracking.

6. The computer implemented method of motion capture of claim 1 , wherein deforming the 3D model in the virtual 3D space comprises solving for control values for an animation rig that defines non-linear relationships between at least some of the control values.

7. The computer implemented method of motion capture of claim 6 , wherein the animation rig comprises one or more selections from the group consisting of: a facial rig, a body rig, and a blend shape.

8. The computer implemented method of motion capture of claim 1 , wherein identifying the change to the 2D representation of the subject comprises identifying subsections of textures and tracking 2D location changes of the subsections in a first frame relative to a second frame.

9. The computer implemented method of motion capture of claim 1 , further comprising:
accessing one or more reference frames in the plurality of frames, each reference frame showing the subject; and
tracking 2D changes in a second frame relative to a first frame includes using a reference frame as a key frame specifying texture locations in a future frame during the motion capture.

10. The computer implemented method of motion capture of claim 1 , wherein identifying a change to the 2D representation of the subject between two or more of the plurality of frames comprises:
identifying a location of an occlusion object that indicates an occlusion of textures in a frame; and
ignoring textures in the frame that overlap a projection of the occlusion object in the frame.

11. The computer implemented method of motion capture of claim 1 , further comprising:
reading a plurality of first camera parameters for a first frame in the plurality of frames, the first camera parameters corresponding to a physical state of a real camera when the real camera recorded the first frame; and
determining a plurality of second camera parameters for a second frame in the plurality of frames, the second camera parameters corresponding to a physical state of the real camera when the real camera recorded the second frame.

12. The computer implemented method of motion capture of claim 1 , further comprising:
determining a freeform deformation to be applied to the 3D model; and
adding the freeform deformation to the 3D model.

13. A motion capture system, the system comprising:
a geometry tracker configured to:
access a 3D model;
associate the 3D model with a 2D representation of a subject in a plurality of frames, wherein the plurality of frames captures the subject in a real 3D space; and
deform the 3D model in a virtual 3D space, wherein the deforming is based on:
an identified change to the position of the 2D representation; and
at least one constraint restricting how the 3D model can be deformed, wherein the at least one constraint comprises one or more motion vectors; and

a texture tracker configured to:
identifying the one or more motion vectors derived from the change to the position of the 2D representation of the subject between two or more of the plurality of frames.

14. The motion capture system of claim 13 , wherein each frame in the plurality of frames comprises a 2D image of the subject in a real 3D space.

15. The motion capture system of claim 13 , wherein the geometry tracker is further configured to project the identified change onto the virtual 3D space as 3D constraints.

16. The motion capture system of claim 13 , wherein identifying the change to the 2D representation of the subject comprises identifying subsections of textures and tracking 2D location changes of the subsections in a first frame relative to a second frame.

17. A non-transitory computer readable medium comprising instructions stored thereon which, when executed by one or more processing devices, cause the one or more processing devices to perform operations comprising:
accessing a 3D model;
associating the 3D model with a 2D representation of a subject in a plurality of frames, wherein the plurality of frames captures the subject in a real 3D space;
identifying one or more motion vectors derived from a change to the position of the 2D representation of the subject between two or more of the plurality of frames; and
deforming the 3D model in a virtual 3D space, wherein the deforming is based on:
the identified change to the position of the 2D representation; and
at least one constraint restricting how the 3D model can be deformed,

wherein the at least one constraint comprises the one or more motion vectors.

18. The non-transitory computer readable medium of claim 17 , wherein each frame in the plurality of frames comprises a 2D image of the subject in a real 3D space.

19. The non-transitory computer readable medium of claim 17 , wherein the instructions cause the one or more processing devices to perform operations further comprising projecting the identified change onto the virtual 3D space as 3D constraints.

20. The non-transitory computer readable medium of claim 17 , wherein identifying the change to the 2D representation of the subject comprises identifying subsections of textures and tracking 2D location changes of the subsections in a first frame relative to a second frame.

Assignees

• Lucasfilm Entertainment Co Ltd

Inventors

• Mallet Ronald
• Bhat Kiran S
• Wooley Kevin

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06K9/00 Primary

  Physics · mapped topic

• G06T7/251 Primary

  involving models · CPC title