Flow-guided motion retargeting

Patent metadata

Field	Value
Publication number	US-12056792-B2
Application number	US-202117557834-A
Country	US
Kind code	B2
Filing date	Dec 21, 2021
Priority date	Dec 30, 2020
Publication date	Aug 6, 2024
Grant date	Aug 6, 2024

Abstract

Systems and methods herein describe a motion retargeting system. The motion retargeting system accesses a plurality of two-dimensional images comprising a person performing a plurality of body poses, extracts a plurality of implicit volumetric representations from the plurality of body poses, generates a three-dimensional warping field, the three-dimensional warping field configured to warp the plurality of implicit volumetric representations from a canonical pose to a target pose, and based on the three-dimensional warping field, generates a two-dimensional image of an artificial person performing the target pose.

First claim

What is claimed is:

1. A method comprising:
accessing a plurality of two-dimensional images comprising a person performing a plurality of body poses;
extracting, using a first neural network, a plurality of implicit volumetric representations from the plurality of body poses;
generating a three-dimensional warping field, the three-dimensional warping field configured to warp the plurality of implicit volumetric representations from a canonical pose to a target pose; and
based on the three-dimensional warping field, generating a two-dimensional image of an artificial person performing the target pose.

2. The method of claim 1 , wherein each implicit volumetric representation of the plurality of volumetric representations is in a unique resolution.

3. The method of claim 1 wherein accessing the plurality of two-dimensional images further comprises:
for each image of the plurality of two-dimensional images:
extracting a foreground region using a pre-trained segmentation model.

4. The method of claim 1 , wherein the first neural network comprises a two-dimensional encoding network and a three-dimensional encoding network.

5. The method of claim 1 , wherein the generated two-dimensional image is generated by a second neural network.

6. The method of claim 5 , wherein the second neural network comprises a two-dimensional decoding network and a three-dimensional decoding network.

7. The method of claim 1 , wherein generating the two-dimensional image further comprises:
applying an image transformation to the two-dimensional image.

8. The method of claim 7 , wherein the image transformation is a rotation.

9. A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to perform operations comprising:
accessing a plurality of two-dimensional images comprising a person performing a plurality of body poses;
extracting, using a first neural network, a plurality of implicit volumetric representations from the plurality of body poses;
generating a three-dimensional warping field, the three-dimensional warping field configured to warp the plurality of implicit volumetric representations from a canonical pose to a target pose; and
based on the three-dimensional warping field, generating a two-dimensional image of an artificial person performing the target pose.

10. The system of claim 9 , wherein each implicit volumetric representation of the plurality of volumetric representations is in a unique resolution.

11. The system of claim 9 wherein accessing the plurality of two-dimensional images further comprises:
for each image of the plurality of two-dimensional images:
extracting a foreground region using a pre-trained segmentation model.

12. The system of claim 9 , wherein the first neural network comprises a two-dimensional encoding network and a three-dimensional encoding network.

13. The system of claim 9 , wherein the generated two-dimensional image is generated by a second neural network.

14. The system of claim 13 , wherein the second neural network comprises a two-dimensional decoding network and a three-dimensional decoding network.

15. The system of claim 9 , wherein generating the two-dimensional image further comprises:
applying an image transformation to the two-dimensional image.

16. The system of claim 15 , wherein the image transformation is a rotation.

17. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
accessing a plurality of two-dimensional images comprising a person performing a plurality of body poses;
extracting, using a first neural network, a plurality of implicit volumetric representations from the plurality of body poses;
generating a three-dimensional warping field, the three-dimensional warping field configured to warp the plurality of implicit volumetric representations from a canonical pose to a target pose; and
based on the three-dimensional warping field, generating a two-dimensional image of an artificial person performing the target pose.

18. The computer-readable storage medium of claim 17 , wherein each implicit volumetric representation of the plurality of volumetric representations is in a unique resolution.

19. The computer-readable storage medium of claim 17 wherein accessing the plurality of two-dimensional images further comprises:
for each image of the plurality of two-dimensional images:
extracting a foreground region using a pre-trained segmentation model.

20. The computer-readable storage medium of claim 17 , wherein the first neural network comprises a two-dimensional encoding network and a three-dimensional encoding network.

Assignees

• Snap Inc

Inventors

• Ren Jian
• Chai Menglei
• Woodford Oliver
• Olszewski Kyle
• Tulyakov Sergey

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title