Motion model refinement based on contact analysis and optimization

Patent metadata

Field	Value
Publication number	US-11238634-B2
Application number	US-202016860411-A
Country	US
Kind code	B2
Filing date	Apr 28, 2020
Priority date	Apr 28, 2020
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

In some embodiments, a motion model refinement system receives an input video depicting a human character and an initial motion model describing motions of individual joint points of the human character in a three-dimensional space. The motion model refinement system identifies foot joint points of the human character that are in contact with a ground plane using a trained contact estimation model. The motion model refinement system determines the ground plane based on the foot joint points and the initial motion model and constructs an optimization problem for refining the initial motion model. The optimization problem minimizes the difference between the refined motion model and the initial motion model under a set of plausibility constraints including constraints on the contact foot joint points and a time-dependent inertia tensor-based constraint. The motion model refinement system obtains the refined motion model by solving the optimization problem.

First claim

The invention claimed is:

1. A computer-implemented method in which one or more processing devices perform operations comprising:
determining contact joint points for a human character contained in a sequence of video frames using a trained contact estimation model, wherein the contact joint points are determined by the trained contact estimation model as being in contact with a ground plane;
determining the ground plane, in a three-dimensional space defined by an initial motion model of the human character, based on the contact joint points and the initial motion model, wherein the initial motion model of the human character is estimated from the sequence of video frames and describes motions of joint points of the human character in the three-dimensional space;
determining a refined motion model over the initial motion model by performing an optimization under a set of constraints defined based on the initial motion model, the determined ground plane, and the contact joint points; and
applying the refined motion model on a target computer-generated character.

2. The computer-implemented method of claim 1 , wherein performing the optimization comprises minimizing a difference between the refined motion model and the initial motion model, and wherein the set of constraints comprise one or more of a time-dependent inertia tensor-based constraint, a leg-length constraint, a foot-length constraint, or constraints on the contact joint points.

3. The computer-implemented method of claim 1 , wherein performing the optimization comprises determining a position of a center of mass (COM) of the human character.

4. The computer-implemented method of claim 1 , further comprising, prior to performing the optimization, smoothing the motions of the joint points in the initial motion model along a time dimension by applying a low-pass filter on the joint points of the initial motion model over time.

5. The computer-implemented method of claim 1 , further comprising:
prior to performing the optimization, re-targeting the initial motion model on the target computer-generated character to generate a re-targeted initial motion model, wherein the refined motion model is generated by performing the optimization based on the re-targeted initial motion model.

6. The computer-implemented method of claim 1 , wherein the contact joint points comprise contact foot joint points, and wherein the trained contact estimation model is configured to determine, based on at least a portion of joint points of the human character, a contact label for each of foot joint points of the human character, the contact label indicating whether an associated joint point is in contact with the ground plane or not.

7. The computer-implemented method of claim 6 , wherein the foot joint points comprise a left heel joint point, a left toe joint point, a right heel joint point, and a right toe joint point, and wherein the portion of joint points used for determining the contact labels comprise joint points on a lower body of the human character.

8. A system comprising:
a processing device; and
a non-transitory computer-readable medium having program code that is stored thereon, the program code executable by the processing device for performing operations, comprising:
generating an input vector for a sequence of video frames by determining positions of joint points of a character contained in the sequence of video frames;
executing a trained neural network based on the input vector to generate indications of contact statuses of a subset of the joint points with a ground plane;
estimating the ground plane in a three-dimensional space based on joint points that are in contact with the ground plane according to the indications of the contact statuses and an initial motion model of the character in the three-dimensional space extracted from the sequence of video frames;
determining a refined motion model based on the initial motion model by performing an optimization under a set of constraints; and
applying the refined motion model on a target computer-generated character to generate a computer animated character.

9. The system of claim 8 , wherein the joint points of the character comprise joint points of a lower body of the character, and wherein the subset of the joint points comprise a joint point of a left toe of the character, a joint point of a left heel of the character, a joint point of a right toe of the character, and a joint point of a right heel of the character.

10. The system of claim 8 , wherein the trained neural network comprises:
an input layer configured for receiving an input vector containing positions of joint points of a character contained in a first set of video frames; and
an output layer configured for outputting, for each joint point in a subset of the joint points of the character in a second set of video frames, an indication of a contact status of the joint point with a ground plane, wherein the first set of video frames comprises the second set of video frames.

11. The system of claim 8 , wherein performing the optimization comprises minimizing a difference between the refined motion model and the initial motion model, and wherein the set of constraints comprise one or more of a time-dependent inertia tensor-based constraint, a leg-length constraint, a foot-length constraint, or constraints on the subset of the joint points based on their respective contact statuses.

12. The system of claim 8 , wherein performing the optimization comprises determining a position of a center of mass (COM) of the character and positions of the subset of the joint points in the refined motion model.

13. The system of claim 8 , wherein the character is a human character.

14. A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:
determining contact joint points for a human character contained in a sequence of video frames using a trained contact estimation model, wherein the contact joint points are determined by the trained contact estimation model as being in contact with a ground plane;
determining the ground plane, in a three-dimensional space defined by an initial motion model of the human character, based on the contact joint points and the initial motion model, wherein the initial motion model of the human character is estimated from the sequence of video frames and describes motions of joint points of the human character in the three-dimensional space;
determining a refined motion model over the initial motion model by performing an optimization under a set of constraints defined based on the initial motion model, the determined ground plane, and the contact joint points; and
applying the refined motion model on a target computer-generated character.

15. The non-transitory computer-readable medium of claim 14 , wherein performing the optimization comprises minimizing a difference between the refined motion model and the initial motion model, and wherein the set of constraints comprise one or more of a time-dependent inertia tensor-based constraint, a leg-length constraint, a foot-length constraint, or constraints on the contact joint points.

16. The non-transitory computer-readable medium of claim 14 , wherein performing the optimization comprises determining a position of a center of mass (COM) of the human character.

17. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise, prior to performing the optimization, smoothing the mo

Assignees

• Adobe Inc

Inventors

• Yang Jimei
• Rempe Davis
• Russell Bryan
• Hertzmann Aaron

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/251

  involving models · CPC title