Generating enhanced three-dimensional object reconstruction models from sparse set of object images

Patent metadata

Field	Value
Publication number	US-11669986-B2
Application number	US-202117233122-A
Country	US
Kind code	B2
Filing date	Apr 16, 2021
Priority date	Apr 16, 2021
Publication date	Jun 6, 2023
Grant date	Jun 6, 2023

Abstract

Enhanced methods and systems for generating both a geometry model and an optical-reflectance model (an object reconstruction model) for a physical object, based on a sparse set of images of the object under a sparse set of viewpoints. The geometry model is a mesh model that includes a set of vertices representing the object's surface. The reflectance model is SVBRDF that is parameterized via multiple channels (e.g., diffuse albedo, surface-roughness, specular albedo, and surface-normals). For each vertex of the geometry model, the reflectance model includes a value for each of the multiple channels. The object reconstruction model is employed to render graphical representations of a virtualized object (a VO based on the physical object) within a computation-based (e.g., a virtual or immersive) environment. Via the reconstruction model, the VO may be rendered from arbitrary viewpoints and under arbitrary lighting conditions.

First claim

What is claimed is:

1. A method for a reconstruction of a three-dimensional ( 3 D) physical object, the method comprising:
receiving a set of images, wherein each image of the set of images depicts the physical object from a separate viewpoint of a set of viewpoints;
generating, based on the set of image, a set of per-view (PV) reflectance maps, a set of composite feature maps, and a coarse geometry model encoding a coarse representation of a geometry of a surface of the physical object, wherein each PV reflectance map of the set of PV reflectance maps corresponds to one of the viewpoints of the set of viewpoints and each composite feature map of the set of composite feature maps corresponds to one of the viewpoints of the set of viewpoints and is based a composite of the set of images warped to the corresponding viewpoint;
aggregating the set of PV reflectance maps across the set of viewpoints to generate a coarse reflectance model that includes a set of estimated reflectance parameters for each vertex of the coarse geometry model and for each viewpoint of the set of viewpoints; and
jointly refining, based on the set of composite feature maps, the coarse geometry model and the coarse reflectance model to generate a refined geometry model and a refined reflectance model, wherein the refined geometry model encodes a refined representation of the geometry and the refined reflectance models encodes a refined representation of the reflectance property, wherein the coarse geometry model and the coarse reflectance model are refined by fusing the set of estimated reflectance parameters for each vertex to reconstruct geometry and reflectance associated with the physical object.

2. The method for claim 1 , further comprising:
generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
aggregating the set of PV depth maps across the set of viewpoints to generate the set of PV reflectance maps and the set of composite feature maps; and
employing the set of PV depth maps to generate the coarse geometry model.

3. The method of claim 2 , further comprising:
generating a set of PV feature maps based on the set of images, wherein each PV feature map of the set of PV feature maps corresponds to one of the viewpoints of the set of viewpoints;
for each viewpoint of the set of viewpoints, warping each PV feature map to each of the other viewpoints of the set of viewpoints to generate a set of warped feature maps; and
generating the set of PV depth maps based on the set of warped feature maps.

4. The method of claim 2 , further comprising:
generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV diffuse albedo maps based on the set of PV depth maps, wherein each PV diffuse albedo map of the set of PV diffuse albedo maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV surface-roughness maps based on the set of PV depth maps, wherein each PV surface-roughness map of the set of PV surface-roughness maps corresponds to one of the viewpoints of the set of viewpoints;
generating a set of PV specular albedo maps based on the set of PV depth maps, wherein each PV specular map of the set of PV specular maps corresponds to one of the viewpoints of the set of viewpoints; and
generating the set of PV reflectance maps based on a combination of the set of PV surface-normal maps, the set of PV diffuse albedo maps, the set of PV surface-roughness maps, and the set of PV specular albedo maps.

5. The method of claim 2 , further comprising:
generating an intermediate feature map for each pair of viewpoints of the set of viewpoints; and
generating each composite feature map of the set of composite feature maps by aggregating the intermediate feature maps for each pair of viewpoints that includes a viewpoint of the pair of viewpoints.

6. The method of claim 1 , further comprising:
generating a synthetic image for each image of the set of input images based on the coarse geometry model, the coarse reflectance model, and the set of composite feature maps; and
determining an error function based on a comparison of each of the synthetic images with the corresponding image of the set of images; and
refining each of the coarse geometry model and the coarse reflectance model based on the error function.

7. A computing system for a reconstruction of a three-dimensional (3D) physical object, comprising:
a processor device; and
a computer-readable storage medium, coupled with the processor device, having instructions stored thereon, which, when executed by the processor device, perform actions comprising:
steps for generating a set of per-view (PV) reflectance maps based on a set of images, wherein each image of the set of images depicts the physical object from a separate viewpoint of a set of viewpoints and each PV reflectance map of the set of PV reflectance maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of composite feature maps, wherein each composite feature map of the set of composite feature maps corresponds to one of the viewpoints of the set of viewpoints and is based on a composite of the set of images warped to the corresponding viewpoint;

steps for jointly refining, based on the set of composite feature maps, a coarse geometry model and a coarse reflectance model to generate a refined geometry model and a refined reflectance model, wherein the refined geometry model encodes a refined representation of the geometry and the refined reflectance models encodes a refined representation of the reflectance property, and wherein the coarse geometry model and the coarse reflectance model are refined by fusing a set of estimated reflectance parameters for each vertex to reconstruct geometry and reflectance.

8. The computing system of claim 7 , wherein the actions further comprise:
steps for generating a set of PV depth maps based on the set of images, wherein each PV depth map of the set of PV depth maps corresponds to one of the viewpoints of the set of viewpoints;
steps for aggregating the set of PV depth maps across the set of viewpoints to generate the set of PV reflectance maps and the set of composite feature maps; and
steps for employing the set of PV depth maps to generate the coarse geometry model.

9. The computing system of claim 8 , wherein the actions further comprise:
steps for generating a set of PV feature maps based on the set of images, wherein each PV feature maps of the set of PV feature maps corresponds to one of the viewpoints of the set of viewpoints;
for each viewpoint of the set of viewpoints, steps for warping each PV feature map to each of the other viewpoints of the set of viewpoints to generate a set of warped feature maps; and
steps for generating the set of PV depth maps based on the set of warped feature maps.

10. The computing system of claim 8 , wherein the actions further comprise:
steps for generating a set of PV surface-normal maps based on the set of PV depth maps, wherein each PV surface-normal map of the set of PV surface-normal maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of PV diffuse albedo maps based on the set of PV depth maps, wherein each PV diffuse albedo map of the set of PV diffuse albedo maps corresponds to one of the viewpoints of the set of viewpoints;
steps for generating a set of PV surface-roughness maps based on the set of PV depth maps, wher

Assignees

• Adobe Inc
• Univ California

Inventors

• Bi sai
• Xu Zexiang
• Sunkavalli Kalyan Krishna
• Kriegman David Jay
• Ramamoorthi Ravi

Classifications

• G06T7/514 Primary

  from specularities · CPC title

• G06T7/596

  from three or more stereo images · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title