System and method for generating character poses using deep learning

Patent metadata

Field	Value
Publication number	US-12254550-B2
Application number	US-202318201041-A
Country	US
Kind code	B2
Filing date	May 23, 2023
Priority date	Sep 4, 2020
Publication date	Mar 18, 2025
Grant date	Mar 18, 2025

Abstract

A method of generating or modifying poses in an animation of a character are disclosed. Variable numbers and types of supplied inputs are combined into a single input. The variable numbers and types of supplied inputs correspond to one or more effector constraints for one or more joints of the character. The single input is transformed into a pose embedding. The pose embedding includes a machine-learned representation of the single input. The pose embedding is expanded into a pose representation output. The pose representation output includes local rotation data and global position data for the one or more joints of the character.

First claim

We claim:

1. A system comprising:
one or more computer processors;
one or more computer memories;
a set of instructions stored in the one or more computer memories, the set of instructions configuring the one or more computer processors to perform operations, the operations comprising:
combining a set of zero or more inputs including one or more different effector types for each of one or more joints of a character into a single input;
transforming the single input into a pose embedding, the pose embedding including a machine-learned representation of the single input; and
expanding the pose embedding into a pose representation output, the pose representation output including rotation data and position data for the one or more joints of the character.

2. The system of claim 1 , wherein the expanding of the pose embedding into the pose representation output includes translating the pose embedding into predictions of 3D joint positions for the one or more joints, the translating using a global position decoder machine-learned model, and generating the rotation data for the one or more joints, the generating using an inverse kinematic decoder machine-learned model.

3. The system of claim 2 , wherein one or more of the global position decoder machine-learned model and the inverse kinematic decoder machine-learned model are trained using a variable number and type of input effector constraints.

4. The system of claim 1 , the operations further comprising:
transforming the rotation data by applying a forward kinematics pass to the pose representation output, the transforming including using one or more skeleton kinematic equations.

5. The system of claim 4 , wherein the transforming includes using one or more neural networks that are programmable for use with each of variable numbers and types of the set of zero or more inputs without requiring retraining of the one or more neural networks.

6. The system of claim 1 , wherein the transforming of the single input is performed by a pose encoder machine-learned model, wherein the pose encoder machine-learned model is trained using a variable number and type of input effector constraints.

7. The system of claim 1 , wherein the transforming includes applying a machine-learned model that, during training, uses a combination loss function, the combination loss function including rotation and position error terms via randomized weights based on randomly-generated effector tolerance levels.

8. The system of claim 1 , wherein a first joint of the one or more joints is constrained by 3D position coordinates and global rotation and a second joint of the one or more joints is constrained with a look-at effector.

9. A non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, causes the one or more computer processors to perform operations, the operations comprising:
combining a set of zero or more inputs including one or more different effector types for each of one or more joints of a character into a single input;
transforming the single input into a pose embedding, the pose embedding including a machine-learned representation of the single input; and
expanding the pose embedding into a pose representation output, the pose representation output including rotation data and position data for the one or more joints of the character.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the expanding of the pose embedding into the pose representation output includes translating the pose embedding into predictions of 3D joint positions for the one or more joints, the translating using a global position decoder machine-learned model, and generating the rotation data for the one or more joints, the generating using an inverse kinematic decoder machine-learned model.

11. The non-transitory computer-readable storage medium of claim 10 , wherein one or more of the global position decoder machine-learned model and the inverse kinematic decoder machine-learned model are trained using a variable number and type of input effector constraints.

12. The non-transitory computer-readable storage medium of claim 9 , the operations further comprising:
transforming the rotation data by applying a forward kinematics pass to the pose representation output, the transforming including using one or more skeleton kinematic equations.

13. The non-transitory computer-readable storage medium of claim 9 , wherein the transforming includes using one or more neural networks that are programmable for use with each of variable numbers and types of the set of zero or more inputs without requiring retraining of the one or more neural networks.

14. The non-transitory computer-readable storage medium of claim 9 , wherein the transforming of the single input is performed by a pose encoder machine-learned model, wherein the pose encoder machine-learned model is trained using a variable number and type of input effector constraints.

15. The non-transitory computer-readable storage medium of claim 9 , wherein the transforming includes applying a machine-learned model that, during training, uses a combination loss function, the combination loss function including rotation and position error terms via randomized weights based on randomly-generated effector tolerance levels.

16. A method comprising:
combining a set of zero or more inputs including one or more different effector types for each of one or more joints of a character into a single input;
transforming the single input into a pose embedding, the pose embedding including a machine-learned representation of the single input; and
expanding the pose embedding into a pose representation output, the pose representation output including rotation data and position data for the one or more joints of the character.

17. The method of claim 16 , wherein the expanding of the pose embedding into the pose representation output includes translating the pose embedding into predictions of 3D joint positions for the one or more joints, the translating using a global position decoder machine-learned model, and generating the rotation data for the one or more joints, the generating using an inverse kinematic decoder machine-learned model.

18. The method of claim 17 , wherein one or more of the global position decoder machine-learned model and the inverse kinematic decoder machine-learned model are trained using a variable number and type of input effector constraints.

19. The method of claim 16 , further comprising:
transforming the rotation data by applying a forward kinematics pass to the pose representation output, the transforming including using one or more skeleton kinematic equations.

20. The method of claim 19 , wherein the transforming includes using one or more neural networks that are programmable for use with each of variable numbers and types of the set of zero or more inputs without requiring retraining of the one or more neural networks.

Assignees

• Unity IPR ApS

Inventors

• Bocquelet Florent Benjamin
• Laflamme Dominic
• Oreshkin Boris

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/20081

  Training; Learning · CPC title