Creating simulation-ready clothed 3d human avatars from text descriptions

Patent metadata

Field	Value
Publication number	US-2025363703-A1
Application number	US-202418979205-A
Country	US
Kind code	A1
Filing date	Dec 12, 2024
Priority date	May 21, 2024
Publication date	Nov 27, 2025
Grant date	—

Abstract

Apparatuses, systems, and techniques for generating a clothed three-dimensional (3D) avatar character from a text prompt and enabling smooth animation through physics or neural simulators. In at least one embodiment, a clothed 3D avatar is generated through body layer modeling and garment layer modeling based on text descriptions. The outputs from the body layer and garment layer modeling are combined to generate an animation-ready, clothed 3D avatar.

First claim

What is claimed is:

1 . A computer-implemented method for generating a clothed three-dimensional (3D) avatar, comprising:
receiving one or more text prompts; modeling, based on the text prompts, a body layer for the clothed 3D avatar; modeling, based on the text prompts, a garment layer for the clothed 3D avatar; combining the body layer and the garment layer for the clothed 3D avatar; and outputting the clothed 3D avatar.

2 . The method of claim 1 , wherein modeling the body layer for the clothed 3D avatar comprises:
determining, based on the one or more text prompts, a body mesh for the clothed 3D avatar; modeling a first set of Gaussians corresponding to the body mesh; and outputting a body model of the clothed 3D avatar.

3 . The method of claim 1 , wherein modeling the garment layer for the clothed 3D avatar comprises:
determining, based on the one or more text prompts, a latent code corresponding to a garment template; generating, based on the latent code and using a decoder, an unsigned distance field (UDF) corresponding to the garment layer of the clothed 3D avatar; modeling a second set of Gaussians corresponding to the UDF; and outputting a garment model of the clothed 3D avatar.

4 . The method of claim 3 , wherein the latent code among a plurality of latent codes, wherein the plurality of latent codes correspond to a plurality of garment templates, and wherein the plurality of latent codes and the plurality of garment templates are learned from a garment dataset.

5 . The method of claim 3 , wherein the UDF comprises a plurality of unsigned distances, the method further comprises:
determining, for each unsigned distance of the plurality of unsigned distances, an opacity value for the respective unsigned distance.

6 . The method of claim 3 , further comprising:
deriving a garment mesh based on the UDF; generating, based on the garment mesh, a sequence of garment meshes corresponding to a sequence of poses associated with the clothed 3D avatar; and generating a third set of Gaussians corresponding to the garment mesh by transforming the second set of Gaussians corresponding to the UDF.

7 . The method of claim 3 , further comprising:
training the generation of latent code and 3D Gaussians, wherein the training comprises:
sampling a fourth set of 3D Gaussians from a pre-defined garment template mesh to optimize the fourth set of 3D Gaussians and a predicted latent code, wherein the predicted latent code corresponds to the pre-defined garment template mesh;
extracting a predicted garment mesh from the optimized latent code; and
sampling a fifth set of 3D Gaussians from the predicted garment mesh to optimize the attributes of the fifth set of 3D Gaussians.

8 . The method of claim 1 , further comprising:
modeling lighting effects on the garment layer for the clothed 3D avatar.

9 . The method of claim 1 , wherein modeling the body layer and modeling the garment layer are performed in parallel, in sequence, or in a hybrid manner.

10 . The method of claim 1 , wherein the clothed 3D avatar is a clothed 3D human avatar.

11 . A system for generating a clothed three-dimensional (3D) avatar comprising:
one or more processors configured to: receive one or more text prompts; model, based on the text prompts, a body layer for the clothed 3D avatar; model, based on the text prompts, a garment layer for the clothed 3D avatar; combine the body layer and the garment layer for the clothed 3D avatar; and output the clothed 3D avatar.

12 . The system of claim 11 , wherein modeling the body layer for the clothed 3D avatar comprises:
determining, based on the one or more text prompts, a body mesh for the clothed 3D avatar; modeling a first set of Gaussians corresponding to the body mesh; and outputting a body model of the clothed 3D avatar.

13 . The system of claim 11 , wherein modeling the garment layer for the clothed 3D avatar comprises:
determining, based on the one or more text prompts, a latent code corresponding to a garment template; generating, based on the latent code and using a decoder, an unsigned distance field (UDF) corresponding to the garment layer of the clothed 3D avatar; modeling a second set of Gaussians corresponding to the UDF; and outputting a garment model of the clothed 3D avatar.

14 . The system of claim 13 , wherein the latent code among a plurality of latent codes, wherein the plurality of latent codes correspond to a plurality of garment templates, and wherein the plurality of latent codes and the plurality of garment templates are learned from a garment dataset.

15 . The system of claim 13 , wherein the UDF comprises a plurality of unsigned distances, and wherein the one or more processors are further configured to:
determine, for each unsigned distance of the plurality of unsigned distances, an opacity value for the respective unsigned distance.

16 . The system of claim 13 , wherein the one or more processors are further configured to:
derive a garment mesh based on the UDF; generate, based on the garment mesh, a sequence of garment meshes corresponding to a sequence of poses associated with the clothed 3D avatar; and generate a third set of Gaussians corresponding to the garment mesh by transforming the second set of Gaussians corresponding to the UDF.

17 . The system of claim 13 , wherein the one or more processors are further configured to:
train the generation of latent code and 3D Gaussians, wherein the training comprises:
sampling a fourth set of 3D Gaussians from a pre-defined garment template mesh to optimize the fourth set of 3D Gaussians and a predicted latent code, wherein the predicted latent code corresponds to the pre-defined garment template mesh;
extracting a predicted garment mesh from the optimized latent code; and
sampling a fifth set of 3D Gaussians from the predicted garment mesh to optimize the attributes of the fifth set of 3D Gaussians.

18 . The system of claim 11 , wherein the one or more processors are further configured to:
model lighting effects on the garment layer for the clothed 3D avatar.

19 . The system of claim 11 , wherein modeling the body layer and modeling the garment layer are performed in parallel, in sequence, or in a hybrid manner.

20 . The system of claim 11 , wherein the clothed 3D avatar is a clothed 3D human avatar.

21 . A machine-readable medium for generating a clothed three-dimensional (3D) avatar, having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to:
receive one or more text prompts; model, based on the text prompts, a body layer for an avatar corresponding to the clothed 3D avatar; model, based on the text prompts, a garment layer for the clothed 3D avatar; combine the body layer and the garment layer for the clothed 3D avatar; and output the clothed 3D avatar.

22 . The machine-readable medium of claim 19 ,
wherein modeling the body layer for the clothed 3D avatar comprises:
determining, based on the one or more text prompts, a body mesh for the clothed 3D avatar;
modeling a first set of Gaussians corresponding to the body mesh; and
outputting a body model of the clothed 3D avatar, and
wherein modeling the garment layer for the clothed 3D avatar comprises:
determining, based on the one or more text prompts, a latent code corresponding to a garment template;
generating, based on the latent code and

Assignees

• Nvidia Corp

Inventors

• Li Xueting
• Yuan Ye
• Iqbal Umar
• Macklin Miles
• Leaf Jonathan
• Xiang Donglai
• De Mello Shalini
• Kautz Jan

Classifications

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06F40/00

  Handling natural language data (speech analysis or synthesis, speech recognition G10L) · CPC title

• G06T2210/16

  Cloth · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title