Three-dimensional model recovery from two-dimensional images

Patent metadata

Field	Value
Publication number	US-11734890-B2
Application number	US-202117175792-A
Country	US
Kind code	B2
Filing date	Feb 15, 2021
Priority date	Aug 17, 2020
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

A three-dimensional (3D) model of an object is recovered from two-dimensional (2D) images of the object. Each image in the set of 2D images includes the object captured from a different camera position and deformations of a base mesh that defines the 3D model may be computed corresponding to each image. The 3D model may also include a texture map that represents the lighting and material properties of the 3D model. Recovery of the 3D model relies on analytic antialiasing to provide a link between pixel colors in the 2D images and geometry of the 3D model. A modular differentiable renderer design yields high performance by leveraging existing, highly optimized hardware graphics pipelines to reconstruct the 3D model. The differential renderer renders images of the 3D model and differences between the rendered images and reference images are propagated backwards through the rendering pipeline to iteratively adjust the 3D model.

First claim

What is claimed is:

1. A computer-implemented method for constructing a three-dimensional (3D) model, comprising:
rendering an image of the 3D model defined by a base mesh and an initial surface texture representing material properties of the 3D model, wherein an antialiasing operation performed by a rendering pipeline comprises
computing an intersection point on a segment intersected by an edge of rasterized geometry that covers at least a portion of the adjacent pixels, wherein the segment connects a first sample location within a first pixel of the image and a second sample location within a second pixel of the image that is adjacent to the first pixel, and
shading the adjacent pixels based on the intersection point;

computing pixel differences based on the image and a reference image;
propagating the pixel differences backwards through the antialiasing operation of the rendering pipeline to compute geometry changes and surface texture changes corresponding to reducing the pixel differences;
adjusting the initial surface texture based on the surface texture changes to produce a modified surface texture for the 3D model; and
adjusting weights used to deform the base mesh based on the geometry changes, wherein the weights are applied to the base mesh by a deformation network to produce geometry for the 3D model.

2. The computer-implemented method of claim 1 , wherein the geometry is defined by vertex locations.

3. The computer-implemented method of claim 1 , further comprising repeating the rendering, computing, propagating, and adjusting for at least one additional reference image.

4. The computer-implemented method of claim 3 , wherein the reference image and the at least one additional reference image are each associated with a different camera position.

5. The computer-implemented method of claim 1 , wherein the initial surface texture is a uniform color.

6. The computer-implemented method of claim 5 , wherein the surface texture represents lighting of the 3D model.

7. The computer-implemented method of claim 1 , wherein the edge forms a silhouette and the antialiasing operation further comprises identifying the rasterized geometry associated with a first winding and a second geometry associated with the first winding.

8. The computer-implemented method of claim 1 , wherein the antialiasing operation approximates a pixel integral based on a location of a silhouette edge within the adjacent pixels.

9. The computer-implemented method of claim 1 , wherein at least one of the steps of rendering, computing, propagating, and adjusting are performed within a cloud computing environment.

10. The computer-implemented method of claim 1 , wherein at least one of the steps of rendering, computing, propagating, and adjusting are performed on a server or in a data center to generate the image and the 3D model, and at least one of the image and the 3D model is streamed to a user device.

11. The computer-implemented method of claim 1 , wherein at least one of the steps of rendering, computing, propagating, and adjusting are performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

12. The computer-implemented method of claim 1 , wherein at least one of the steps of rendering, computing, propagating, and adjusting is performed on a virtual machine comprising a portion of a graphics processing unit.

13. The computer-implemented method of claim 1 , wherein the antialiasing operation further comprises storing results of the antialiasing operation to compute the geometry changes during the propagating.

14. A system, comprising:
processor configured to:
rendering an image of the 3D model defined by a base mesh and an initial surface texture representing material properties of the 3D model, wherein an antialiasing operation performed by a rendering pipeline comprises
computing an intersection point on a segment connecting sample locations within adjacent pixels of the image and an edge of rasterized geometry that covers at least a portion of the adjacent pixels, and
shading the adjacent pixels based on the intersection point;

computing pixel differences based on the image and a reference image;
propagating the pixel differences backwards through the antialiasing operation of the rendering pipeline to compute geometry changes corresponding to reducing the pixel differences and surface texture changes corresponding to reducing the pixel differences;
adjusting the initial surface texture based on the surface texture changes to produce a modified surface texture for the 3D model; and
adjusting weights used to deform the base mesh based on the geometry changes, wherein the weights are applied to the base mesh by a deformation network to produce geometry for the 3D model.

15. The system of claim 14 , wherein the initial geometry is defined by vertex locations.

16. The system of claim 14 , wherein the initial geometry is defined by weights applied to a base mesh.

17. The system of claim 14 , wherein the processor is configured to repeat the rendering, computing, propagating, and adjusting for at least one additional reference image.

18. The system of claim 14 , wherein the initial surface texture is a uniform color.

19. A non-transitory computer-readable media storing computer instructions for three-dimensional (3D) model construction that, when executed by one or more processors, cause the one or more processors to perform the steps of:
rendering an image of the 3D model defined by a base mesh and an initial surface texture representing material properties of the 3D model, wherein an antialiasing operation performed by a rendering pipeline comprises
computing an intersection point on a segment connecting sample locations within adjacent pixels of the image and an edge of rasterized geometry that covers at least a portion of the adjacent pixels, and
shading the adjacent pixels based on the intersection point;

computing pixel differences based on the image and a reference image;
propagating the pixel differences backwards through the antialiasing operation of the rendering pipeline to compute geometry changes corresponding to reducing the pixel differences and surface texture changes corresponding to reducing the pixel differences;
adjusting the initial surface texture based on the surface texture changes to produce a modified surface texture for the 3D model;
adjusting weights used to deform the base mesh based on the geometry changes, wherein the weights are applied to the base mesh by a deformation network to produce geometry for the 3D model.

20. The non-transitory computer-readable media of claim 19 , wherein the initial surface texture is a uniform color.

Assignees

• Nvidia Corp

Inventors

• Laine Samuli Matias
• Hellsten Janne Johannes
• Karras Tero Tapani
• Seol Yeongho
• Lehtinen Jaakko T
• Aila Timo Oskari

Classifications

• G06T5/80

  Geometric correction · CPC title

• G06T17/205 Primary

  Re-meshing · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title

• G06T15/50

  Lighting effects · CPC title