Rendering textured surface using surface-rendering neural networks

Patent metadata

Field	Value
Publication number	US-11816779-B2
Application number	US-202117538311-A
Country	US
Kind code	B2
Filing date	Nov 30, 2021
Priority date	Nov 30, 2021
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

Methods and systems disclosed herein relate generally to surface-rendering neural networks to represent and render a variety of material appearances (e.g., textured surfaces) at different scales. The system includes receiving image metadata for a texel that includes position, incoming and outgoing radiance direction, and a kernel size. The system applies a offset-prediction neural network to the query to identify an offset coordinate for the texel. The system inputs the offset coordinate to a data structure to determine a feature vector for the texel of the textured surface. The reflectance feature vector is then processed using a decoder neural network to estimate a light-reflectance value of the texel, at which the light-reflectance value is used to render the texel of the textured surface.

First claim

What is claimed is:

1. A method comprising:
receiving, by a surface-rendering application, image metadata for rendering a texel of a textured surface, wherein the image metadata includes a coordinate identifying a location of the texel, an incoming radiance direction, an outgoing radiance direction, and a kernel size;
estimating, by the surface-rendering application and based on the image metadata, a light-reflectance value for the texel, wherein estimating the light-reflectance value includes:
applying an offset-prediction neural network to the coordinate and the incoming radiance direction to predict an offset coordinate of the texel;
inputting the offset coordinate to a data structure to determine a reflectance feature vector of the texel at the kernel size, wherein: (i) the data structure includes a set of neural textures; (ii) each neural texture of the set of neural textures includes a set of reflectance feature vectors at a respective kernel size; and (iii) each reflectance feature vector of the set of reflectance feature vectors includes one or more latent variables that encode visual characteristics of a respective texel; and
applying a decoder neural network to the reflectance feature vector, the incoming radiance direction, and the outgoing radiance direction to estimate a light-reflectance value for the texel; and
rendering, by the surface-rendering application and based on the light-reflectance value, the texel of the textured surface at the location, wherein the texel depicts at least part of the textured surface at an image scale that corresponds to the kernel size.

2. The method of claim 1 , wherein the data structure is a neural-texture pyramid, and wherein values of the one or more latent variables of a reflectance feature vector are learned using a backpropagation algorithm.

3. The method of claim 1 , wherein applying the offset-prediction neural network to the coordinate and the incoming radiance direction further includes:
querying a neural-offset texture using the coordinate to identify an offset feature vector of the coordinate;
applying the offset-prediction neural network to the offset feature vector and the incoming radiance direction to estimate a ray depth associated with the coordinate; and
applying an offset function to the ray depth to predict the offset coordinate.

4. The method of claim 1 , wherein querying the data structure further includes:
identifying, from a first neural texture of the set of neural textures, a first location within the first neural texture based on bilinear interpolation of the offset coordinate;
selecting a second neural texture of the set of neural textures based on the kernel size;
interpolating the first location of the first neural texture to a second location within the second neural texture; and
selecting a reflectance feature vector at the second location as the reflectance feature vector that represents the texel.

5. The method of claim 1 , wherein the offset-prediction neural network is trained using an unsupervised training algorithm.

6. The method of claim 1 , wherein the decoder neural network includes a multi-layer perceptron network.

7. The method of claim 1 , wherein the offset-prediction neural network and the decoder neural network were trained together using a backpropagation algorithm.

8. The method of claim 1 , wherein the decoder neural network was trained separately from a training of the offset-prediction neural network.

9. A system comprising:
a neural offset module configured to:
receive image metadata for rendering a texel of a textured surface, wherein the image metadata includes a coordinate identifying a location of the texel, an incoming radiance direction, an outgoing radiance direction, and a kernel size; and
apply an offset-prediction neural network to the coordinate and the incoming radiance direction to predict an offset coordinate of the texel;
a neural-texture query module configured to:
receive, from the neural offset module, the offset coordinate of the texel; and inputting the offset coordinate to a data structure to determine a reflectance feature vector of the texel at the kernel size, wherein: (i) the data structure includes a set of neural textures; (ii) each neural texture of the set of neural textures includes a set of reflectance feature vectors at a respective kernel size; and (iii) each reflectance feature vector of the set of reflectance feature vectors includes one or more latent variables that encode visual characteristics of a respective texel;
a decoder configured to:
identify, from the image metadata, the incoming radiance direction and the outgoing radiance direction;
receive, from the neural-texture query module, the reflectance feature vector of the texel; and
apply a decoder neural network to the reflectance feature vector, the incoming radiance direction, and the outgoing radiance direction to estimate a light- 24 reflectance value for the texel; and
a rendering engine configured to render, based on the light-reflectance value, the texel of the textured surface at the location, wherein the texel depicts at least part of the textured surface at an image scale that corresponds to the kernel size.

10. The system of claim 9 , wherein the data structure is a neural-texture pyramid, and wherein values of the one or more latent variables of a reflectance feature vector are learned using a backpropagation algorithm.

11. The system of claim 9 , wherein the neural offset module is further configured to:
querying a neural-offset texture using the coordinate to identify an offset feature vector of the coordinate;
applying the offset-prediction neural network to the offset feature vector and the incoming radiance direction to estimate a ray depth associated with the coordinate; and
applying an offset function to the ray depth to predict the offset coordinate.

12. The system of claim 9 , wherein the neural-texture query module is further configured to:
identifying, from a first neural texture of the set of neural textures, a first location within the first neural texture based on bilinear interpolation of the offset coordinate;
selecting a second neural texture of the set of neural textures based on the kernel size;
interpolating the first location of the first neural texture to a second location within the second neural texture; and
selecting a reflectance feature vector at the second location as the reflectance feature vector that represents the texel.

13. The system of claim 9 , wherein the offset-prediction neural network is trained using an unsupervised training algorithm.

14. The system of claim 9 , wherein the decoder neural network includes a multi-layer perceptron network.

15. The system of claim 9 , wherein the offset-prediction neural network and the decoder neural network were trained together using a backpropagation algorithm.

16. The system of claim 9 , wherein the decoder neural network was trained separately from a training of the offset-prediction neural network.

17. A non-transitory computer-readable storage medium including instructions configured to cause one or more data processors to perform actions including:
receiving image metadata for rendering a texel of a textured surface, wherein the image metadata includes a coordinate identifying a location of the texel, an incoming radiance direction, an outgoing radiance direction, and a kernel size;
a step for estimating a light-reflectance value for the texel by: (i) applying an offset-prediction neural network to the coordinate and the incoming radiance direction to pr

Assignees

• Adobe Inc
• Univ California

Inventors

• Lakshminarayana Krishna Bhargava Mullia
• Xu Zexiang
• Hasan Milos
• Ramamoorthi Ravi
• Kuznetsov Alexandr

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06T15/55

  Radiosity · CPC title

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title