Shape and appearance reconstruction with deep geometric refinement

Patent metadata

Field	Value
Publication number	US-2023252714-A1
Application number	US-202217669053-A
Country	US
Kind code	A1
Filing date	Feb 10, 2022
Priority date	Feb 10, 2022
Publication date	Aug 10, 2023
Grant date	—

Abstract

One embodiment of the present invention sets forth a technique for performing shape and appearance reconstruction. The technique includes generating a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image. The technique also includes producing, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings. The technique further includes generating an updated reconstruction of the object based on the first set of corrections.

First claim

What is claimed is:

1 . A computer-implemented method for performing shape and appearance reconstruction, the computer-implemented method comprising:
generating a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image; producing, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings; and generating an updated reconstruction of the object based on the first set of corrections.

2 . The computer-implemented method of claim 1 , further comprising training the neural network based on a training dataset that includes a set of target corrections.

3 . The computer-implemented method of claim 1 , further comprising:
producing, via the neural network, a second set of corrections associated with the at least a portion of the set of parameters based on a second target image of the object and a second set of renderings associated with the object, wherein generating the updated reconstruction is further based on an aggregation of the first set of corrections and the second set of corrections.

4 . The computer-implemented method of claim 3 , wherein the aggregation comprises a weighted combination of the first set of corrections and the second set of corrections, and wherein the weighted combination is generated based on a first set of visibilities associated with the first set of corrections and a second set of visibilities associated with the second set of corrections.

5 . The computer-implemented method of claim 1 , wherein generating the updated reconstruction comprises:
converting the first set of corrections into a second set of corrections in a canonical space associated with the updated reconstruction; and generating the updated reconstruction in the canonical space based on the second set of corrections.

6 . The computer-implemented method of claim 1 , wherein the first set of corrections comprises a set of offsets to a set of coordinates included in the first set of renderings.

7 . The computer-implemented method of claim 6 , wherein the set of coordinates comprises at least one of a spatial coordinate or a texture coordinate.

8 . The computer-implemented method of claim 1 , wherein the first set of renderings comprises at least one of a vertex coordinate rendering, a texture coordinate rendering, an albedo rendering, or a surface normal rendering.

9 . The computer-implemented method of claim 1 , further comprising generating the set of parameters based on a loss between the first target image and a rendered image that is generated based on the set of parameters.

10 . The computer-implemented method of claim 1 , wherein the set of parameters comprises at least one of an identity parameter, an expression parameter, a geometry parameter, an albedo parameter, a pose parameter, or a lighting parameter.

11 . One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
generating a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image; producing, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings; and generating an updated reconstruction of the object based on the first set of corrections.

12 . The one or more non-transitory computer readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the step of training the neural network based on an L1 loss between the first set of corrections and a corresponding set of target corrections and a gradient loss between the first set of corrections and the corresponding set of target corrections.

13 . The one or more non-transitory computer readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the steps of:
producing, via the neural network, a second set of corrections associated with the at least a portion of the set of parameters based on a second target image of the object and a second set of renderings associated with the object, wherein generating the updated reconstruction is further based on an aggregation of the first set of corrections and the second set of corrections.

14 . The one or more non-transitory computer readable media of claim 13 , wherein the aggregation is generated based on a first set of visibilities associated with the first set of corrections and a second set of visibilities associated with the second set of corrections.

15 . The one or more non-transitory computer readable media of claim 11 , wherein generating the reconstruction comprises:
converting the first set of corrections into a second set of corrections in a canonical space associated with the updated reconstruction; and generating the updated reconstruction in the canonical space based on the second set of corrections and a coarse reconstruction associated with the set of parameters.

16 . The one or more non-transitory computer readable media of claim 11 , wherein the first set of corrections comprises a set of offsets to a set of coordinates included in the first set of renderings.

17 . The one or more non-transitory computer readable media of claim 16 , wherein the set of coordinates comprises at least one of a spatial coordinate associated with a geometry for the object or a texture coordinate associated with a texture for the object.

18 . The one or more non-transitory computer readable media of claim 11 , wherein the set of parameters is associated with a latent space of one or more decoder neural networks.

19 . The one or more non-transitory computer readable media of claim 11 , wherein the neural network comprises a convolutional encoder that performs downsampling of a first set of feature maps associated with the first target image and the first set of renderings and a convolutional decoder that performs upsampling of a second set of feature maps associated with target image and the first set of renderings.

20 . A system, comprising:
one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and,
when executing the instructions, are configured to:
generate a first set of renderings associated with an object based on a set of parameters that represent a reconstruction of the object in a first target image;
produce, via a neural network, a first set of corrections associated with at least a portion of the set of parameters based on the first target image and the first set of renderings; and
generate an updated reconstruction of the object based on the first set of corrections.

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Bradley Derek Edward
• Chandran Prashanth
• Urnau Gotardo Paulo Fabiano
• Otto Christopher Andreas
• Serifi Agon
• Zoss Gaspard

Classifications

• G06T15/04 Primary

  Texture mapping · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T15/205

  Image-based rendering · CPC title

• G06V40/169

  Holistic features and representations, i.e. based on the facial image taken as a whole · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title