Neural animation layering for synthesizing movement

Patent metadata

Field	Value
Publication number	US-2024257429-A1
Application number	US-202318520344-A
Country	US
Kind code	A1
Filing date	Nov 27, 2023
Priority date	Jan 26, 2021
Publication date	Aug 1, 2024
Grant date	—

Abstract

In some embodiments, the dynamic animation generation system can provide a deep learning framework to produce a large variety of martial arts movements in a controllable manner from unstructured motion capture data. The system can imitate animation layering using neural networks with the aim to overcome challenges when mixing, blending and editing movements from unaligned motion sources. The system can synthesize movements from given reference motions and simple user controls, and generate unseen sequences of locomotion, but also reconstruct signature motions of different fighters. For achieving this task, the dynamic animation generation system can adopt a modular framework that is composed of the motion generator, that maps the trajectories of a number of key joints and root trajectory to the full body motion, and a set of different control modules that map the user inputs to such trajectories.

First claim

What is claimed is:

1 . A computer-implemented method for dynamically generating animation of a virtual character performing certain actions in a virtual environment of an instance of a video game, the method comprising:
receiving a current frame of a virtual character within a virtual environment of an instance of a video game, wherein the current frame includes current pose data for the virtual character; identifying a plurality of possible behaviors for the virtual character for the next frame based on the current pose data in the current frame, wherein the next frame is a subsequent frame to the current frame; receiving, from a user of the video game, an input to perform at least a first and second behavior of the plurality of possible behaviors; determining a plurality of pose data for the first and second behavior; performing layering of the plurality of pose data corresponding to the first and second behavior on the current pose data to generate layered data; applying the layered data to a gating network to generate weights; and applying the weights to a pose predictor network configured to generate next pose data for the next frame.

2 . The computer-implemented method of claim 1 , wherein the gating network receives velocity magnitudes of future joint trajectories for the next pose data, wherein the weights generated by the gating network are blended weights of the future joint trajectories.

3 . The computer-implemented method of claim 1 , wherein performing layering comprises applying additive layering to the pose data corresponding to the first and second behavior.

4 . The computer-implemented method of claim 1 , wherein performing layering comprises applying override layering to the pose data corresponding to the first and second behavior.

5 . The computer-implemented method of claim 1 , wherein performing layering comprises applying blend layering to the pose data corresponding to the first and second behavior.

6 . The computer-implemented method of claim 1 , wherein the pose predictor network blends weights of a fixed number of structurally identical networks.

7 . The computer-implemented method of claim 6 , wherein applying the layered data to the gating network comprises applying velocity magnitudes of future joint trajectories for the layered data, wherein generated weights by the gating network comprises blended weights dictating the influence of each of the structurally identical networks.

8 . The computer-implemented method of claim 1 , wherein the method further comprises applying the current frame with current pose data to the pose predictor network, wherein the pose predictor network is configured to generate next pose data for the next frame based on the current pose data.

9 . A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a current frame of a virtual character within a virtual environment of an instance of a video game, wherein the current frame includes current pose data for the virtual character; identifying a plurality of possible behaviors for the virtual character for the next frame based on the current pose data in the current frame, wherein the next frame is a subsequent frame to the current frame; receiving, from a user of the video game, an input to perform at least a first and second behavior of the plurality of possible behaviors; determining a plurality of pose data for the first and second behavior; performing layering of the plurality of pose data corresponding to the first and second behavior on the current pose data to generate layered data; applying the layered data to a gating network to generate weights; and applying the weights to a pose predictor network configured to generate next pose data for the next frame.

10 . The system of claim 9 , wherein the next pose data for the next frame does not match pose data previously stored by the system.

11 . The system of claim 9 , wherein the gating network applies gating variables according to the following:

X
i

=

{

C

i
+
1

,

P
i

,

g
i

}

.

12 . The system of claim 9 , wherein the pose predictor network generates the next pose data according to the following:

(

C

i
+
1

,

P
i

)

→

P

i
+
1

.

13 . The system of claim 9 , wherein the operations further comprise:
mapping trajectories of a number of key joints and a root trajectory of the virtual character in the current frame, wherein the plurality of possible behaviors are identified based on the mapped trajectories and root trajectory.

14 . The system of claim 9 , wherein the plurality of possible behaviors for the virtual character for the next frame are identified by a neural network configured to determine possible behaviors for the virtual character for the next frame based on the current frame.

15 . The system of claim 9 , wherein the plurality of possible behaviors for the virtual character for the next frame are identified by a motion matching system for the virtual character in the instance of the video game.

16 . A non-transitory computer storage media storing instructions that when executed by a system of one or more processors, cause the one or more processors to perform operations comprising:
receiving a current frame of a virtual character within a virtual environment of an instance of a video game, wherein the current frame includes current pose data for the virtual character; identifying a plurality of possible behaviors for the virtual character for the next frame based on the current pose data in the current frame, wherein the next frame is a subsequent frame to the current frame; receiving, from a user of the video game, an input to perform at least a first and second behavior of the plurality of possible behaviors; determining a plurality of pose data for the first and second behavior; performing layering of the plurality of pose data corresponding to the first and second behavior on the current pose data to generate layer

Assignees

• Electronic Arts Inc

Inventors

• Starke Wolfram Sebastian
• Zhao Yiwei
• Sardari Mohsen
• Chaput Harold Henry
• Aghdaie Navid

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• A63F13/573

  using trajectories of game objects, e.g. of a golf ball according to the point of impact · CPC title

• A63F2300/646

  for calculating the trajectory of an object · CPC title

• A63F2300/6607

  for animating game characters, e.g. skeleton kinematics · CPC title

• G06N3/02

  Neural networks · CPC title