Systems and methods for reconstructing body shape and pose

Patent metadata

Field	Value
Publication number	US-11908071-B2
Application number	US-202117495960-A
Country	US
Kind code	B2
Filing date	Oct 7, 2021
Priority date	Oct 7, 2021
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

The present disclosure is generally directed to reconstructing representations of bodies from images. An example method of the present disclosure includes inputting, into a machine-learned reconstruction model, input data descriptive of an image depicting a body; predicting, using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and outputting, using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations. In the example method, one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between relaxed-constraint representations generated from a prior set of surface marker locations predicted according to the one or more parameters and parametric representations generated from the prior set using kinematic constraints associated with the body.

First claim

What is claimed is:

1. A computer-implemented method for reconstructing representations of bodies from images, the method comprising:
inputting, by one or more computing devices into a machine-learned reconstruction model, input data descriptive of an image depicting a body;
predicting, by the one or more computing devices and using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and
outputting, by the one or more computing devices and using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations;
wherein one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between (i) a relaxed-constraint representation generated from a prior set of surface marker locations predicted according to the one or more parameters and (ii) a parametric representation generated from the prior set using kinematic constraints associated with the body.

2. The computer-implemented method of claim 1 , wherein the body is a human body and the kinematic constraints correspond to anthropometric constraints.

3. The computer-implemented method of claim 1 , wherein the output representation is a relaxed-constraint representation.

4. The computer-implemented method of claim 1 , wherein the marker prediction component comprises one or more encoder layers.

5. The computer-implemented method of claim 4 , wherein the one or more encoder layers respectively comprise self-attention models.

6. The computer-implemented method of claim 5 , wherein predicting the set of surface marker locations comprises:
encoding, by the one or more computing devices and using the one or more encoder layers, a surface marker embedding along with the input data; and
updating, by the one or more computing devices, the set of surface marker locations based at least in part on the encoded surface marker embedding.

7. The computer-implemented method of claim 6 , wherein an output of each of the one or more encoder layers is used to iteratively refine the set of surface marker locations, the output corresponding to the surface marker embedding.

8. The computer-implemented method of claim 7 , wherein the one or more encoder layers comprise a plurality of encoder layers that share one or more machine-learned weights.

9. The computer-implemented method of claim 1 , comprising:
transforming, by the one or more computing devices and using a capture model, the input data; and
wherein the output representation is obtained in a capture space corresponding to the capture model.

10. The computer-implemented method of claim 9 , wherein the capture model is based at least in part on a perspective model.

11. A system for reconstructing representations of bodies from images, comprising:
one or more processors; and
one or more memory devices storing computer-readable instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising:
inputting, into a machine-learned reconstruction model, input data descriptive of an image depicting a body;
predicting, using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and
outputting, using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations;
wherein one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between (i) a relaxed-constraint representation generated from a prior set of surface marker locations predicted according to the one or more parameters and (ii) a parametric representation generated from the prior set using kinematic constraints associated with the body.

12. The system of claim 11 , wherein the body is a human body and the kinematic constraints correspond to anthropometric constraints.

13. The system of claim 11 , wherein the output representation is a relaxed- constraint representation.

14. The system of claim 11 , wherein the marker prediction component comprises one or more encoder layers.

15. The system of claim 14 , wherein the one or more encoder layers respectively comprise self-attention models.

16. The system of claim 15 , wherein predicting the set of surface marker locations comprises:
encoding, using the one or more encoder layers, a surface marker embedding along with the input data; and
updating the set of surface marker locations based at least in part on the encoded surface marker embedding.

17. The system of claim 16 , wherein an output of each of the one or more encoder layers is used to iteratively refine the set of surface marker locations, the output corresponding to the surface marker embedding.

18. The system of claim 17 , wherein the one or more encoder layers comprise a plurality of encoder layers that share one or more machine-learned weights.

19. A system for reconstructing representations of bodies from images, comprising:
one or more processors; and
one or more memory devices storing computer-readable instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising:
inputting, into a machine-learned marker prediction model, input data descriptive of an image depicting a body;
predicting, using the marker prediction model, a set of surface marker locations on the body;
outputting, using a machine-learned marker poser model, a parametric representation of the body that corresponds to the set of surface marker locations; and
updating one or more parameters of the marker prediction model based at least in part on a consistency loss corresponding to a distance between the parametric representation and a relaxed-constraint representation associated with the predicted set of surface marker locations.

20. The system of claim 19 , wherein the output representation is a relaxed-constraint representation.

Assignees

• Google Llc

Inventors

• Sminchisescu Cristian
• Zanfir Mihai
• Zanfir Andrei
• Bazavan Eduard Gabriel
• Freeman William Tafel
• Sukthankar Rahul

Classifications

• G06T12/00

  Tomographic reconstruction from projections · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T11/003

  Physics · mapped topic