Motion representations for articulated animation

Patent metadata

Field	Value
Publication number	US-11836835-B2
Application number	US-202117364218-A
Country	US
Kind code	B2
Filing date	Jun 30, 2021
Priority date	Jun 30, 2020
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

Systems and methods herein describe novel motion representations for animating articulated objects consisting of distinct parts. The described systems and method access source image data, identify driving image data to modify image feature data in the source image sequence data, generate, using an image transformation neural network, modified source image data comprising a plurality of modified source images depicting modified versions of the image feature data, the image transformation neural network being trained to identify, for each image in the source image data, a driving image from the driving image data, the identified driving image being implemented by the image transformation neural network to modify a corresponding source image in the source image data using motion estimation differences between the identified driving image and the corresponding source image, and stores the modified source image data.

First claim

What is claimed is:

1. A method comprising:
accessing, by a computing device, source image data comprising a plurality of source images;
identifying driving image data to modify image feature data in the source image data;
generating, using an image transformation neural network, modified source image data comprising a plurality of modified source images depicting modified versions of the image feature data,
the image transformation neural network being trained to identify, for each image in the source image data, a driving image from the driving image data, the identified driving image being implemented by the image transformation neural network to modify a corresponding source image in the source image data using motion estimation differences between the identified driving image and the corresponding source image, the image transformation neural network comprising a background motion detector neural network configured to predict an affine background transformation based on the source image data and the driving image data; and
storing the modified source image data on the computing device.

2. The method of claim 1 , wherein the driving image data comprises an ordered set of driving images.

3. The method of claim 1 , wherein the image transformation neural network comprises a region predictor neural network and a generation neural network.

4. The method of claim 3 , wherein the region predictor neural network is configured to generate motion estimation differences between the identified driving image and the corresponding source image.

5. The method of claim 4 , further comprising:
generating a set of source image heatmaps from the source image data;
generating a set of driving image heatmaps from the driving image data;
computing a set of source principal axes for the set of source image heatmaps;
computing a set of driving principal axes for the set of driving image heatmaps; and
generating a transformation of the source image data based on the computed set of source principal axes and the set of driving principal axes.

6. The method of claim 3 , wherein the generation neural network is configured to generate each modified source image in the modified source image data by applying the motion estimation differences between the identified driving image and the corresponding source image from the source image data to the corresponding source image.

7. A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to:
access source image data comprising a plurality of source images;
identify driving image data to modify image feature data in the source image data;
generate, using an image transformation neural network, modified source image data comprising a plurality of modified source images depicting modified versions of the image feature data,
the image transformation neural network being trained to identify, for each image in the source image data, a driving image from the driving image data, the identified driving image being implemented by the image transformation neural network to modify a corresponding source image in the source image data using motion estimation differences between the identified driving image and the corresponding source image, the image transformation neural network comprising a background motion detector neural network configured to predict an affine background transformation based on the source image data and the driving image data; and
store the modified source image data.

8. The system of claim 7 , wherein the driving image data comprises an ordered set of driving images.

9. The system of claim 7 , wherein the image transformation neural network comprises a region predictor neural network and a generation neural network.

10. The system of claim 9 , wherein the region predictor neural network is configured to generate motion estimation differences between the identified driving image and the corresponding source image.

11. The system of claim 10 , wherein the instructions further configure the system to:
generate a set of source image heatmaps from the source image data;
generate a set of driving image heatmaps from the driving image data;
compute a set of source principal axes for the set of source image heatmaps;
compute a set of driving principal axes for the set of driving image heatmaps; and
generate a transformation of the source image data based on the computed set of source principal axes and the set of driving principal axes.

12. The system of claim 9 , wherein the generation neural network is configured to generate each modified source image in the modified source image data by applying the motion estimation differences between the identified driving image and the corresponding source image from the source image data to the corresponding source image.

13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
access source image data comprising a plurality of source images;
identify driving image data to modify image feature data in the source image data;
generate, using an image transformation neural network, modified source image data comprising a plurality of modified source images depicting modified versions of the image feature data,
the image transformation neural network being trained to identify, for each image in the source image data, a driving image from the driving image data, the identified driving image being implemented by the image transformation neural network to modify a corresponding source image in the source image data using motion estimation differences between the identified driving image and the corresponding source image, the image transformation neural network comprising a background motion detector neural network configured to predict an affine background transformation based on the source image data and the driving image data; and
store the modified source image data.

14. The computer-readable storage medium of claim 13 , wherein the driving image data comprises an ordered set of driving images.

15. The computer-readable storage medium of claim 13 , wherein the image transformation neural network comprises a region predictor neural network and a generation neural network.

16. The computer-readable storage medium of claim 15 , wherein the region predictor neural network is configured to generate motion estimation differences between the identified driving image and the corresponding source image.

17. The computer-readable storage medium of claim 16 , wherein the instructions further configure the computer to:
generate a set of source image heatmaps from the source image data;
generate a set of driving image heatmaps from the driving image data;
compute a set of source principal axes for the set of source image heatmaps;
compute a set of driving principal axes for the set of driving image heatmaps; and
generate a transformation of the source image data based on the computed set of source principal axes and the set of driving principal axes.

Assignees

• Snap Inc

Inventors

• Chai Menglei
• Ren Jian
• Siarohin Aliaksandr
• Tulyakov Sergey
• Woodford Oliver

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title