Methods of 3d clothed human reconstruction and animation from monocular image

Patent metadata

Field	Value
Publication number	US-2023281921-A1
Application number	US-202217982945-A
Country	US
Kind code	A1
Filing date	Nov 8, 2022
Priority date	Mar 1, 2022
Publication date	Sep 7, 2023
Grant date	—

Abstract

A method for 3D human model reconstruction and animation includes receiving a two-dimensional (2D) image of a human, segmenting the 2D image into a foreground with the human and a background without the human in the 2D image, generating a parametric model comprising a pose, a shape, and one or more rigging parameters based on the human in the foreground, predicting a textured three-dimensional (3D) model using implicit surface reconstruction of the human in the foreground, aligning the parametric model and the textured 3D model using a 3D registration, and generating a textured 3D clothed human model based on the aligned parametric model and the predicted textured 3D model.

First claim

What is claimed is:

1 . A method executed by at least one processor, the method comprising:
receiving, by a processor, a two-dimensional (2D) image of a human; segmenting the 2D image into a foreground with the human and a background without the human in the 2D image; generating a parametric model comprising a pose, a shape, and one or more rigging parameters based on the human in the foreground; predicting a textured three-dimensional (3D) model using implicit surface reconstruction of the human in the foreground; aligning the parametric model and the textured 3D model using a 3D registration; and generating a textured 3D clothed human model based on the aligned parametric model and the predicted textured 3D model, wherein the 3D clothed human model comprises at least a 3D shape of the human in the 2D image with reconstructed surface textures in a reconstructed 3D space.

2 . The method of claim 1 , wherein the aligning comprises:
transferring the texture from the textured 3D model to the parametric model; and transferring the pose, the shape, and the one or more rigging parameters from the parametric model to the textured 3D model.

3 . The method of claim 1 , wherein the parametric model is generated to fit the 2D image input.

4 . The method of claim 1 , wherein the parametric model and textured 3D model are aligned by minimizing a difference of one or more distance fields in 3D space.

5 . The method of claim 1 , wherein the textured 3D clothed human model is animated by a motion capture dataset.

6 . The method of claim 1 , wherein the parametric model is based on one of the following statistical human models: SMPL, SMPL-X, or STAR.

7 . The method of claim 1 , wherein the generating the textured 3D model using implicit surface reconstruction is based on a PIFu scheme from the 2D image.

8 . An apparatus comprising:
at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code including: receiving code configured to cause the at least one processor to receive, a two-dimensional (2D) image of a human; segmenting code configured to cause the at least one processor to segment the 2D image into a foreground with the human and a background without the human in the 2D image; first generating code configured to cause the at least one processor to generate a parametric model comprising a pose, a shape, and one or more rigging parameters based on the human in the foreground; second generating code configured to cause the at least one processor to predict a textured three-dimensional (3D) model using implicit surface reconstruction of the human in the foreground; aligning code configured to cause the at least one processor to align the parametric model and the textured 3D model using a 3D registration; and third generating code configured to cause the at least one processor to generate a textured 3D clothed human model based on the aligned parametric model and the predicted textured 3D model, wherein the 3D clothed human model comprises at least a 3D shape of the human in the 2D image with reconstructed surface textures in a reconstructed 3D space.

9 . The apparatus of claim 8 , wherein the aligning code further causes the processor to:
transfer the texture from the textured 3D model to the parametric model; and transfer the pose, shape, and the one or more rigging parameters from the parametric model to the textured 3D model.

10 . The apparatus of claim 8 , wherein the parametric model is generated to fit the 2D image input.

11 . The apparatus of claim 8 , wherein the parametric model and textured 3D model are aligned by minimizing a difference of one or more distance fields in 3D space.

12 . The apparatus of claim 8 , wherein the textured 3D clothed human model is animated by a motion capture dataset.

13 . The apparatus of claim 8 , wherein the parametric model is based on any one of the following statistical human models: SMPL, SMPL-X, or STAR.

14 . The apparatus of claim 8 , wherein the textured 3D model using implicit surface reconstruction is based on a PIFu scheme from the 2D image.

15 . A non-transitory computer readable medium having stored thereon computer code which, when executed by at least one processor, causes the at least one processor to at least:
receive a two-dimensional (2D) image of a human; segment the 2D image into a foreground with the human and a background without the human in the 2D image; generate a parametric model comprising a pose, a shape, and one or more rigging parameters based on the human in the foreground; predict a textured three-dimensional (3D) model using implicit surface reconstruction of the human in the foreground; align the parametric model and the textured 3D model using a 3D registration; and generate a textured 3D clothed human model based on the aligned parametric model and the predicted textured 3D model, wherein the 3D clothed human model comprises at least a 3D shape of the human in the 2D image with reconstructed surface textures in a reconstructed 3D space.

16 . The non-transitory computer readable medium according to claim 15 , wherein the instructions to cause the at least one processor to align the parametric model and the textured 3D model further cause the processor to:
transfer the texture from the textured 3D model to the parametric model; and transfer the pose, the shape, and the one or more rigging parameters from the parametric model to the textured 3D model.

17 . The non-transitory computer readable medium according to claim 15 , wherein the parametric model is generated to fit the 2D image input.

18 . The non-transitory computer readable medium according to claim 15 , wherein the parametric model and textured 3D model are aligned by minimizing a difference of one or more distance fields in 3D space.

19 . The non-transitory computer readable medium according to claim 15 , wherein the textured 3D clothed human model is animated by a motion capture dataset.

20 . The non-transitory computer readable medium according to claim 15 , wherein the parametric model is based on any one of the following statistical human models: SMPL, SMPL-X, or STAR.

Assignees

• Tencent America LLC

Inventors

• Miao Dan
• Jian Bing
• Feng Weiwei
• Liu Shan
• Wang Ye

Classifications

• G06V40/103 Primary

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/75

  involving models · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title