Contact-aware retargeting of motion

Patent metadata

Field	Value
Publication number	US-12033261-B2
Application number	US-202117385559-A
Country	US
Kind code	B2
Filing date	Jul 26, 2021
Priority date	Jul 26, 2021
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

One example method involves a processing device that performs operations that include receiving a request to retarget a source motion into a target object. Operations further include providing the target object to a contact-aware motion retargeting neural network trained to retarget the source motion into the target object. The contact-aware motion retargeting neural network is trained by accessing training data that includes a source object performing the source motion. The contact-aware motion retargeting neural network generates retargeted motion for the target object, based on a self-contact having a pair of input vertices. The retargeted motion is subject to motion constraints that: (i) preserve a relative location of the self-contact and (ii) prevent self-penetration of the target object.

First claim

The invention claimed is:

1. A method performed by one or more computing devices, the method comprising:
receiving source motion for a source object over a time period, the source object characterized by a source object skeleton and a source object geometry, wherein the source object geometry includes multiple source skin vertices, wherein the source motion for the source object comprises a motion of the source object skeleton and a motion of the multiple source skin vertices included in the source object geometry over the time period, and wherein the source motion includes, at a first time frame in the time period, a contact between a first source vertex and a second source vertex of the multiple source skin vertices included in the source object geometry; and
retargeting, via a trained machine-learning model, the source motion to a target object, the target object characterized by a target object skeleton and a target object geometry, wherein the target object geometry is different from the source object geometry, wherein the target object geometry includes multiple target skin vertices that are different from the multiple source skin vertices included in the source object geometry,
wherein the retargeting includes:
calculating an energy function that includes a) a first constraint for preserving a contact between a first target vertex and a second target vertex of the target object geometry in a target motion based upon the contact between the first source vertex and the second source vertex of the source object geometry, and b) a second constraint for reducing self-penetration between the first target vertex and the second target vertex of the target object geometry in the target motion, and
causing, via the retargeting, the target object to have the target motion over the time period that is based upon the source motion, wherein the target motion describes that, at the first time frame in the time period, the first target vertex and the second target vertex of the multiple target skin vertices included in the target object geometry are in contact without any self-penetration of the target object geometry at the first target vertex and the second target vertex.

2. The method of claim 1 , further comprising:
determining that the first source vertex contacts the second source vertex at the first time frame based upon the motion of the multiple source skin vertices included in the source object geometry;
determining that the first target vertex corresponds to the first source vertex and that the second target vertex corresponds to the second source vertex; and
generating the first constraint as a self-contact constraint indicating a self-contact between the first target vertex and the second target vertex,
wherein retargeting the source motion to the target object is based on the self-contact constraint.

3. The method of claim 1 , wherein the retargeting comprises:
responsive to determining the contact between the first source vertex and the second source vertex in the source motion, causing the first target vertex and the second target vertex to have a contact in the target motion.

4. The method of claim 1 , further comprising:
providing the source motion for the source object as input to the trained machine-learning model; and
providing the target object as input to the trained machine-learning model,
wherein the retargeting comprises generating, by the trained machine-learning model, the target motion for the target object based on the source motion for the source object.

5. The method of claim 1 , further comprising training the trained machine-learning model to generate retargeted motion, wherein the training comprises:
for a training source motion and a training target object provided as input to a recurrent neural network (RNN), training the RNN to output a retargeted motion of the training target object by minimizing the energy function, wherein minimizing the energy function includes:
training the first constraint by preserving a self-contact between vertices of a geometry of the training target object in the output retargeted motion based upon an additional self-contact between vertices of a training source geometry associated with the training source motion; and
training the second constraint by reducing self-penetration between the vertices of the geometry of the training target object in the output retargeted motion.

6. The method of claim 5 , wherein the first constraint is characterized by an expression:

1

❘
"\[LeftBracketingBar]"

v

❘
"\[RightBracketingBar]"

⁢

∑

(

i
,
j

)

∈
v



v
i

-

v
j



2

wherein ν i and ν j are vertices of the geometry of the training target object that are to be in contact at a time point in the output retargeted motion.

7. The method of claim 5 , wherein the second constraint is characterized by an expression:

∑

(

r
,
i

)

∈
ℱ

(

∑

v
k

∈

f
r

r
,
i



-

ψ
i

Assignees

• Adobe Inc

Inventors

• Villegas Ruben
• Saito Jun
• Yang Jimei
• Aksit Duygu Ceylan
• Hertzmann Aaron

Classifications

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T9/00

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• G06T7/215

  Motion-based segmentation · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title