Neural networks to render textured materials on curved surfaces

Patent metadata

Field	Value
Publication number	US-12307576-B2
Application number	US-202217993854-A
Country	US
Kind code	B2
Filing date	Nov 23, 2022
Priority date	Nov 23, 2022
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

A scene modeling system accesses a three-dimensional (3D) scene including a 3D object. The scene modeling system applies a silhouette bidirectional texture function (SBTF) model to the 3D object to generate an output image of a textured material rendered as a surface of the 3D object. Applying the SBTF model includes determining a bounding geometry for the surface of the 3D object. Applying the SBTF model includes determining, for each pixel of the output image, a pixel value based on the bounding geometry. The scene modeling system displays, via a user interface, the output image based on the determined pixel values.

First claim

The invention claimed is:

1. A method comprising:
accessing a three-dimensional (3D) scene including one or more 3D objects;
applying a silhouette bidirectional texture function (SBTF) model to a 3D object to generate an output image of a textured material rendered as a surface of the 3D object, wherein applying the SBTF model comprises:
determining a bounding geometry for the surface of the 3D object; and
for each pixel of the output image, determining a pixel value based on the bounding geometry wherein the pixel value comprises a reflectance value, and wherein determining the pixel value for each pixel comprises:
projecting a ray through the pixel into the scene;
determining that the ray encounters the bounding geometry or does not encounter the bounding geometry;
responsive to determining that the ray encounters the bounding geometry, determining the reflectance value for the pixel based in part on a curvature value of the ray, the curvature value filtered by taking a minimum curvature along a curve on the bounding geometry, and applying the pixel to the surface of the 3D object; and

displaying, via a user interface, the output image based on the determined pixel values.

2. The method of claim 1 , wherein the pixel value further comprises an opacity value, and determining the pixel value for each pixel further comprises:
responsive to determining that the ray does not encounter the bounding geometry, assign an opacity value of zero to the pixel.

3. The method of claim 1 , wherein determining the reflectance value further comprises:
determining a UV offset for the pixel based on a location on a surface of the bounding geometry; and
determining the reflectance value based on the location on the surface of the bounding geometry and the UV offset.

4. The method of claim 1 , wherein the SBTF model comprises:
an alpha network configured to determine an opacity value, wherein determining that the ray encounters the bounding geometry comprises determining a non-zero opacity value, wherein determining that the ray does not encounter the bounding geometry comprises determining a zero opacity value.

5. The method of claim 4 , wherein determining the opacity value further comprises:
determining a silhouette cosine value;
if a cosine of the ray is less than the silhouette cosine value, determining a zero value for the opacity value; or
if the cosine of the ray is greater than or equal to the silhouette cosine value, determining a value of one for the opacity value.

6. The method of claim 1 , further comprising training the SBTF model, wherein training the SBTF model comprises:
generating a training dataset of cylindrical patches of varying radii;
applying, to the cylindrical patches of the training dataset, one or more of random rotations, random camera directions, random light directions, or random translations to a UV mapping; and
sampling rays incident upon each of the cylindrical patches from different directions perpendicular to a cylinder axis.

7. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
accessing a three-dimensional (3D) scene including one or more 3D objects;
applying a silhouette bidirectional texture function (SBTF) model to a 3D object to generate an output image of a textured material rendered as a surface of the 3D object, wherein the SBTF model comprises:
an alpha network configured to determine an opacity value for a pixel of the output image based at least in part on a bounding geometry of the surface of the 3D object;
an offset network configured to determine a reflectance value for the pixel wherein the offset network determines the reflectance value for the pixel by:
projecting a ray through the pixel into the scene;
determining the ray encounters the bounding geometry of the surface of the 3D object or does not encounter the bounding geometry of the bounding geometry;
responsive to determining that the ray encounters the bounding geometry, determining the reflectance value for the pixel based in part on a curvature value of the ray, the curvature value filtered by taking a minimum curvature along a curve on the bounding geometry, and applying the pixel to the surface of the 3D object; and

a color network configured to determine a reflectance value for the pixel if the opacity is a non-zero value; and

displaying, via a user interface, the output image based on the determined pixel values.

8. The system of claim 7 , wherein one or more of the alpha network, the offset network, and the color network comprise a fully connected network including a multi-layer perceptron (MLP) that uses a rectified linear unit (ReLU) activation function.

9. The system of claim 7 , wherein the non-zero opacity value indicates that a ray projected through the pixel into the scene encounters the bounding geometry, wherein a zero opacity value indicates that the ray does not encounter the bounding geometry.

10. The system of claim 7 , wherein the color network does not determine the reflectance value if the opacity value is zero.

11. The system of claim 7 , wherein determining the opacity value comprises:
determining a silhouette cosine value;
if a cosine of a ray projected through the pixel into the scene is less than the silhouette cosine value, determining a zero value for the opacity value; or
if the cosine of the ray is greater than or equal to the silhouette cosine value, determining a value of one for the opacity value.

12. The system of claim 7 , wherein the operations further comprise training the SBTF model, wherein training the SBTF model comprises:
generating a training dataset of cylindrical patches of varying radii;
applying, to the cylindrical patches of the training dataset, random rotations and translations to a UV mapping; and
sampling rays incident upon each of the cylindrical patches from different directions perpendicular to a cylinder axis.

13. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
accessing a three-dimensional (3D) scene including one or more 3D objects;
applying a silhouette bidirectional texture function (SBTF) model to a 3D object to generate an output image of a textured material rendered as a surface of the 3D object, wherein applying the SBTF model comprises:
determining a bounding geometry for the surface of the 3D object; and
for each pixel of the output image, determining a silhouette cosine value based on a location on a surface of the bounding geometry of a ray projected through the pixel into the scene;
determine an opacity value based on comparing the silhouette cosine value to a cosine value of the ray; and
determine a pixel value based at least in part on the opacity value and a reflectance value wherein determining the reflectance value comprises:
determining the ray encounters the bounding geometry or does not encounter the bounding geometry;
responsive to determining that the ray encounters the bounding geometry, determining the reflectance value for the pixel based in part on a curvature value of the ray, the curvature value filtered by taking a minimum curvature along a curve on the bounding geometry, and applying the pixel to the surface of the 3D object; and

displaying, via a user interface, the output image based on the determined pixel values.

14. The non-transitory computer-readable medium of claim 13 , wherein the SBTF model determines a zero value for the opacity value if a cosine o

Assignees

• Adobe Inc
• Univ California

Inventors

• Lakshminarayana Krishna Bhargava Mullia
• Xu Zexiang
• Hasan Milos
• Luan fujun
• Kuznetsov Alexandr
• Wang Xuezheng
• Ramamoorthi Ravi

Classifications

• G06T15/04

  Texture mapping · CPC title

• G06T15/506 Primary

  Illumination models · CPC title

• G06T2215/12

  Shadow map, environment map · CPC title

• G06T15/06 Primary

  Ray-tracing · CPC title