Invertible neural skinning

Patent metadata

Field	Value
Publication number	US-12423913-B2
Application number	US-202218090724-A
Country	US
Kind code	B2
Filing date	Dec 29, 2022
Priority date	Dec 29, 2022
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

Invertible Neural Networks (INNs) are used to build an Invertible Neural Skinning (INS) pipeline for reposing characters during animation. A Pose-conditioned Invertible Network (PIN) is built to learn pose-conditioned deformations. The end-to-end Invertible Neural Skinning (INS) pipeline is produced by placing two PINs around a differentiable Linear Blend Skinning (LBS) module using a pose-free canonical representation. The PINs help capture the non-linear surface deformations of clothes across poses and alleviate the volume loss suffered from the LBS operation. Since the canonical representation remains pose-free, the expensive mesh extraction is performed exactly once, and the mesh is reposed by warping it with the learned LBS during an inverse pass through the INS pipeline.

First claim

What is claimed is:

1. An invertible neural skinning (INS) pipeline for animating a three-dimensional (3D) mesh of a deformable object, comprising:
a first trained Pose-conditioned Invertible Neural Network (PIN) that obtains novel poses of the deformable object in a pose-dependent canonical space from a given pose of the deformable object defined by a generic set of bones and the 3D mesh, wherein the first trained PIN comprises an invertible transformation algorithm that provides the novel poses of the deformable object in the pose-dependent canonical space from the given pose provided as input during training;
a differentiable Linear Blend Skinning (LBS) neural network that transforms points in the pose-dependent canonical space to deformed points in novel poses of the deformable object;
a second trained PIN that maps canonical points of the deformable object in the pose-dependent canonical space to canonical points in a pose-independent canonical space; and
a canonical occupancy network or a neural network that receives the canonical points of the deformable object in the pose-independent canonical space,
wherein the given pose of the deformable object is animated, via skeletal bone articulation with the generic set of bones, by extracting a mesh of the deformable object from the canonical occupancy network or the neural network to obtain poses of the deformable object in pose-independent canonical space and reposing mesh vertices of the extracted mesh of the deformable object using the generic set of bones via an inverse pass of the INS pipeline, whereby canonical points in the pose-independent canonical space are mapped by the second trained PIN to pose correspondences of points in pose-dependent canonical space that are applied to the trained differentiable LBS neural network to obtain novel poses of the deformable object that are transformed by the first trained PIN for display as an animated deformable object.

2. The INS pipeline of claim 1 , wherein the first and second PINS are invertible to preserve exact correspondences between inputs and outputs, and the first and second PINS each comprise one-dimensional (1D) and two-dimensional (2D) pose-conditioned coupling layers of an invertible neural network (INN) that are chained together.

3. The INS pipeline of claim 1 , wherein second trained PIN encodes every bone transform in the given pose of the deformable object using an operation map that takes a six-dimensional (6D) input of concatenated three-dimensional (3D) translation and rotation, and obtains pose embedding by concatenating outputs of each bone.

4. A method of animating a three-dimensional (3D) mesh of a deformable object using an invertible neural skinning (INS) pipeline, comprising:
training, by a computer, a first Pose-conditioned Invertible Neural Network (PIN) based on input data including a given pose of the deformable object defined by a generic set of bones and the 3D mesh and an invertible transformation algorithm to generate a first trained PIN for obtaining novel poses of the deformable object in a pose-dependent canonical space;
training, by the computer, a differentiable Linear Blend Skinning (LBS) neural network to transform points in the pose-dependent canonical space to deformed points in novel poses of the deformable object to generate a trained differentiable LBS neural network; and
training, by the computer, a second PIN to map canonical points of the deformable object in the pose-dependent canonical space to canonical points in a pose-independent canonical space to generate a second trained PIN;
passing the canonical points of the deformable object in the pose-independent canonical space to a canonical occupancy network or a neural network; and
animating, via skeletal bone articulation, the given pose of the deformable object with the generic set of bones by extracting a mesh of the deformable object from the canonical occupancy network or the neural network to obtain poses of the deformable object in pose-independent canonical space and reposing mesh vertices of the extracted mesh of the deformable object using the generic set of bones via an inverse pass of the INS pipeline, whereby canonical points in the pose-independent canonical space are mapped by the second trained PIN to pose correspondences of points in pose-dependent canonical space that are applied to the trained differentiable LBS neural network to obtain novel poses of the deformable object that are transformed by the first trained PIN for display as an animated deformable object.

5. The method of claim 4 , further comprising chaining together one-dimensional (1D) and two-dimensional (2D) pose-conditioned coupling layers of an invertible neural network (INN) to form the first and second trained PINS, wherein the first and second trained PINS are invertible to preserve exact correspondences between inputs and outputs.

6. The method of claim 4 , further comprising encoding every bone transform in the given pose of the deformable object using an operation map that takes a six-dimensional (6D) input of concatenated three-dimensional (3D) translation and rotation, and obtaining pose embedding by concatenating outputs of each bone.

7. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor cause the processor to animate a three-dimensional (3D) mesh of a deformable object using an invertible neural skinning (INS) pipeline comprising a first Pose-conditioned Invertible Neural Network (PIN), a differentiable Linear Blend Skinning (LBS) neural network, and a second PIN, by performing operations comprising:
training the first PIN based on input date including a given pose of the deformable object defined by a generic set of bones as input and the 3D mesh and an invertible transformation algorithm to generate a first trained PIN for obtaining novel poses of the deformable object in a pose-dependent canonical space;
training the differentiable LBS neural network to transform points in the pose-dependent canonical space to deformed points in novel poses of the deformable object to generate a trained differentiable LBS neural network;
training the second PIN to map canonical points of the deformable object in the pose-dependent canonical space to canonical points in a pose-independent canonical space to generate a second trained PIN;
passing the canonical points of the deformable object in the pose-independent canonical space to a canonical occupancy network or a neural network; and
animating, via skeletal bone articulation, the given pose of the deformable object with the generic set of bones by extracting a mesh of the deformable object from the canonical occupancy network or the neural network to obtain poses of the deformable object in pose-independent canonical space and reposing mesh vertices of the extracted mesh of the deformable object using the generic set of bones via an inverse pass of the INS pipeline, whereby canonical points in the pose-independent canonical space are mapped by the second trained PIN to pose correspondences of points in pose-dependent canonical space that are applied to the trained differentiable LBS neural network to obtain novel poses of the deformable object that are transformed by the first trained PIN for display as an animated deformable object.

8. The medium of claim 7 , further comprising instructions that when executed by the processor cause the processor to perform operations including encoding every bone transform in the given pose of the deformable object using an operation map that takes a six-dimensional (6D) input of concatenated three-dimensional (3D) translation and rotation, and obtaining pose embedding by concatenating outputs of each

Assignees

• Chai Menglei
• Guler Riza Alp
• Kant Yash Mukund
• Ren Jian
• Siarohin Aliaksandr
• Tulyakov Sergey
• Snap Inc

Inventors

• Chai Menglei
• Guler Riza Alp
• Kant Yash Mukund
• Ren Jian
• Siarohin Aliaksandr
• Tulyakov Sergey

Classifications

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T13/20 Primary

  Three-dimensional [3D] animation · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title