Retargeting skeleton motion sequences through cycle consistency adversarial training of a motion synthesis neural network with a forward kinematics layer

Patent metadata

Field	Value
Publication number	US-10546408-B2
Application number	US-201815926787-A
Country	US
Kind code	B2
Filing date	Mar 20, 2018
Priority date	Mar 20, 2018
Publication date	Jan 28, 2020
Grant date	Jan 28, 2020

Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that use a motion synthesis neural network with a forward kinematics layer to generate a motion sequence for a target skeleton based on an initial motion sequence for an initial skeleton. In certain embodiments, the methods, non-transitory computer readable media, and systems use a motion synthesis neural network comprising an encoder recurrent neural network, a decoder recurrent neural network, and a forward kinematics layer to retarget motion sequences. To train the motion synthesis neural network to retarget such motion sequences, in some implementations, the disclosed methods, non-transitory computer readable media, and systems modify parameters of the motion synthesis neural network based on one or both of an adversarial loss and a cycle consistency loss.

First claim

We claim:

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
input initial joint features for joints of an initial skeleton into a motion synthesis neural network, wherein the initial joint features correspond to an initial time of a motion sequence and the motion synthesis neural network comprises an encoder recurrent neural network, a decoder recurrent neural network, and a forward kinematics layer;
utilize the encoder recurrent neural network and the decoder recurrent neural network to generate predicted joint rotations for joints of a target skeleton differing from the initial skeleton for an initial time of a target motion sequence based on the initial joint features, wherein the target skeleton comprises a segment between a pair of joints differing in length from a corresponding segment in the initial skeleton;
utilize the forward kinematics layer to generate predicted joint features for joints of the target skeleton for the initial time of the target motion sequence based on the predicted joint rotations, the target motion sequence of the target skeleton corresponding to the motion sequence of the initial skeleton; and
based on the predicted joint features, render an animated object performing the target motion sequence of the target skeleton corresponding to the motion sequence of the initial skeleton.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to input the initial joint features for the joints of the initial skeleton into the motion synthesis neural network by inputting positions for the joints of the initial skeleton, a velocity of a root joint of the initial skeleton, and a rotation of the root joint of the initial skeleton into the encoder recurrent neural network.

3. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to utilize the forward kinematics layer to generate the predicted joint features by:
inputting predicted rotation matrices and reference joint positions of the target skeleton into the forward kinematics layer; and
applying a predicted rotation matrix of the predicted rotation matrices to each joint of the target skeleton.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate an encoded feature vector for the initial joint features utilizing the encoder recurrent neural network;
input the encoded feature vector and reference joint positions of the target skeleton into the decoder recurrent neural network; and
generate the predicted joint rotations and a latent feature vector for the initial joint features utilizing the decoder recurrent neural network based on the encoded feature vector and the reference joint positions of the target skeleton.

5. The non-transitory computer readable medium of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
input subsequent joint features for the joints of the initial skeleton and the encoded feature vector for the initial joint features into the motion synthesis neural network, wherein the subsequent joint features correspond to a subsequent time of the motion sequence;
utilize the encoder recurrent neural network and the decoder recurrent neural network to generate subsequent predicted joint rotations for the joints of the target skeleton based on the subsequent joint features and the encoded feature vector for the initial joint features; and
utilize the forward kinematics layer to generate subsequent predicted joint features for joints of the target skeleton for the subsequent time of the motion sequence based on the subsequent predicted joint rotations, wherein the subsequent predicted joint features for joints of the target skeleton reflect the subsequent joint features for the joints of the initial skeleton.

6. The non-transitory computer readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to input the subsequent joint features for the joints of the initial skeleton and the encoded feature vector for the initial joint features into the motion synthesis neural network by inputting subsequent positions for the joints of the initial skeleton, a subsequent velocity of a root joint of the initial skeleton, a subsequent rotation of the root joint of the initial skeleton, and the encoded feature vector for the initial joint features into the encoder recurrent neural network.

7. The non-transitory computer readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to utilize the encoder recurrent neural network and the decoder recurrent neural network to generate the subsequent predicted joint rotations for the joints of the target skeleton by:
generating a subsequent encoded feature vector for the subsequent joint features utilizing the encoder recurrent neural network; and
generating the subsequent predicted joint rotations utilizing the decoder recurrent neural network based on the subsequent encoded feature vector, the predicted joint features for joints of the target skeleton for the initial time of the target motion sequence, the reference joint positions of the target skeleton, and the latent feature vector for the initial joint features.

8. The non-transitory computer readable medium of claim 5 , wherein the target motion sequence performed by the animated object comprises both the predicted joint features and the subsequent predicted joint features.

9. A system for training motion synthesis neural networks to generate retargeted skeleton motion sequences that reflect motion sequences of initial skeletons comprising:
at least one processor;
at least one non-transitory computer memory comprising a motion synthesis neural network that includes an encoder recurrent neural network, a decoder recurrent neural network, and a forward kinematics layer; and
instructions that, when executed by at least one processor, cause the system to:
provide training input joint features for joints of a training initial skeleton to the motion synthesis neural network, wherein the training input joint features correspond to an initial time of a training motion sequence;
utilize the encoder recurrent neural network and the decoder recurrent neural network to generate predicted joint rotations for joints of a training target skeleton differing from the training initial skeleton for an initial time of a training target motion sequence based on the training input joint features, wherein the training target skeleton comprises a segment between a pair of joints differing in length from a corresponding segment in the training initial skeleton;
utilize the forward kinematics layer to generate predicted joint features for joints of the training target skeleton for the initial time of the training target motion sequence based on the predicted joint rotations; and
train the motion synthesis neural network to generate target skeleton motion sequences from initial skeleton motion sequences based on the predicted joint features for the joints of the training target skeleton.

10. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to train the motion synthesis neural network by:
providing the predicted jo

Assignees

• Adobe Inc

Inventors

• Yang Jimei
• Ceylan Duygu
• Villegas Ruben

Classifications

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T2213/12

  Rule based animation · CPC title