Method, device, and computer program product for generating 3D object reconstruction model

Patent metadata

Field	Value
Publication number	US-12548274-B2
Application number	US-202418581670-A
Country	US
Kind code	B2
Filing date	Feb 20, 2024
Priority date	Jan 26, 2024
Publication date	Feb 10, 2026
Grant date	Feb 10, 2026

Abstract

The present disclosure relates to a method, a device, and a computer program product for generating a three-dimensional (3D) object reconstruction model. The method includes acquiring an input image containing a two-dimensional (2D) object. The method further includes determining a shape feature, a texture feature, and a posture feature of the 2D object. The method further includes generating a rendered image based on the shape feature, the texture feature, the posture feature, and the input image. The method further includes generating a 3D object reconstruction model based on the input image and the rendered image, and tuning the 3D object reconstruction model according to a parameter tuning model for stylization. The 3D object reconstruction model generated by this method can realistically reconstruct the 2D object in terms of the shape, the texture, and the posture, so that the 3D object outputted from the model can better match the input image.

First claim

What is claimed is:

1 . A method comprising:
acquiring an input image containing a two-dimensional (2D) object; determining a shape feature, a texture feature, and a posture feature of the 2D object; generating a rendered image based on the shape feature, the texture feature, the posture feature, and the input image; generating a 3D object reconstruction model based on the input image and the rendered image, wherein the 3D object reconstruction model is configured to generate a 3D object from the 2D object utilizing a weighted combination of multiple distinct loss functions including at least one loss function that is generated based on (i) estimated expression coefficients of the 3D object reconstruction model and (ii) an output of an expression classifier of the 3D object reconstruction model; and tuning the 3D object reconstruction model according to a parameter tuning model for stylization.

2 . The method according to claim 1 , wherein determining a shape feature, a texture feature, and a posture feature of the 2D object comprises:
determining an average object shape feature, and an identity feature and an expression feature of the 2D object; and determining the shape feature based on the average object shape feature, the identity feature, and the expression feature as determined.

3 . The method according to claim 1 , wherein determining a shape feature, a texture feature, and a posture feature of the 2D object comprises:
determining an average object texture feature and a constant feature of the 2D object; and determining the texture feature based on the average object texture feature and the constant feature of the 2D object as determined.

4 . The method according to claim 1 , wherein determining a shape feature, a texture feature, and a posture feature of the 2D object comprises:
determining a rotation feature, a scaling feature, and a translation feature of the 2D object; and determining the posture feature based on the rotation feature, the scaling feature, and the translation feature as determined.

5 . The method according to claim 1 , wherein generating the 3D object reconstruction model comprises:
determining a reconstruction loss for measuring a pixel intensity difference, an expression loss for measuring an emotional expression, and a regularization loss for measuring object plausibility based on the shape feature, the texture feature, and the posture feature; and generating the 3D object reconstruction model based on the reconstruction loss, the expression loss, and the regularization loss.

6 . The method according to claim 5 , wherein determining a reconstruction loss for measuring a pixel intensity difference, an expression loss for measuring an emotional expression, and a regularization loss for measuring object plausibility comprises:
determining input features of the input image and rendered features of the rendered image; and generating the reconstruction loss based on the input image, the rendered image, the input features, and the rendered features.

7 . The method according to claim 5 , wherein determining a reconstruction loss for measuring a pixel intensity difference, an expression loss for measuring an emotional expression, and a regularization loss for measuring object plausibility comprises:
generating an emotional expression coefficient based on the shape feature, the texture feature, and the posture feature; extracting a real expression label of the 2D object; and generating the expression loss based on the emotional expression coefficient and the real expression label.

8 . The method according to claim 5 , wherein determining a reconstruction loss for measuring a pixel intensity difference, an expression loss for measuring an emotional expression, and a regularization loss for measuring object plausibility comprises:
determining an average shape feature, an average texture feature, and an average posture feature based on average features of the object; and generating the regularization loss based on the average shape feature, the average texture feature, the average posture feature, the shape feature, the texture feature, and the posture feature.

9 . The method according to claim 1 , wherein tuning the 3D object reconstruction model comprises:
taking the input image, the rendered image, and a style label of the parameter tuning model as observed variables; and tuning the 3D object reconstruction model based on a lower bound of log marginal likelihood of the observed variables.

10 . The method according to claim 1 , further comprising:
reconstructing the 2D object into a 3D object based on the tuned 3D object reconstruction model.

11 . An electronic device, comprising:
at least one processor; and memory coupled to the at least one processor and having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform actions comprising: acquiring an input image containing a two-dimensional (2D) object; determining a shape feature, a texture feature, and a posture feature of the 2D object; generating a rendered image based on the shape feature, the texture feature, the posture feature, and the input image; generating a 3D object reconstruction model based on the input image and the rendered image, wherein the 3D object reconstruction model is configured to generate a 3D object from the 2D object utilizing a weighted combination of multiple distinct loss functions including at least one loss function that is generated based on (i) estimated expression coefficients of the 3D object reconstruction model and (ii) an output of an expression classifier of the 3D object reconstruction model; and tuning the 3D object reconstruction model according to a parameter tuning model for stylization.

12 . The electronic device according to claim 11 , wherein determining a shape feature, a texture feature, and a posture feature of the 2D object comprises:
determining an average object shape feature, and an identity feature and an expression feature of the 2D object; and determining the shape feature based on the average object shape feature, the identity feature, and the expression feature as determined.

13 . The electronic device according to claim 11 , wherein determining a shape feature, a texture feature, and a posture feature of the 2D object comprises:
determining an average object texture feature and a constant feature of the 2D object; and determining the texture feature based on the average object texture feature and the constant feature of the 2D object as determined.

14 . The electronic device according to claim 11 , wherein determining a shape feature, a texture feature, and a posture feature of the 2D object comprises:
determining a rotation feature, a scaling feature, and a translation feature of the 2D object; and determining the posture feature based on the rotation feature, the scaling feature, and the translation feature as determined.

15 . The electronic device according to claim 11 , wherein generating the 3D object reconstruction model comprises:
determining a reconstruction loss for measuring a pixel intensity difference, an expression loss for measuring an emotional expression, and a regularization loss for measuring object plausibility based on the shape feature, the texture feature, and the posture feature; and generating the 3D object reconstruction model based on the reconstruction loss, the expression loss, and the regularization loss.

16 . The electronic device according to claim 15 , wherein de

Assignees

• Dell Products Lp

Inventors

• Wang Zijia
• Liu Zhisong
• Jia Zhen

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title