High-fidelity three-dimensional asset encoding

Patent metadata

Field	Value
Publication number	US-12548241-B2
Application number	US-202318132714-A
Country	US
Kind code	B2
Filing date	Apr 10, 2023
Priority date	Apr 10, 2023
Publication date	Feb 10, 2026
Grant date	Feb 10, 2026

Abstract

Certain aspects and features of this disclosure relate to rendering images by training a neural material and applying the material map to a coarse geometry to provide high-fidelity asset encoding. For example, training can involve sampling for a set of lighting and camera configurations arranged to render an image of a target asset. A value for a loss function comparing the target asset with the neural material can be optimized to train the neural material to encode a high-fidelity model of the target asset. This technique restricts the application of the neural material to a specific predetermined geometry, resulting in a reproducible asset that can be used efficiently. Such an asset can be deployed, as examples, to mobile devices or to the web, where the computational budget is limited, and nevertheless produce highly detailed images.

First claim

What is claimed is:

1 . A method comprising:
accessing a transferable material from a digital asset having an existing geometry; sampling a set of lighting and camera configurations arranged to render an image of a target 3D asset; generating position features for the target 3D asset using a multiresolution hash grid; repeatedly rendering the target 3D asset using the set of lighting and camera configurations; generating training data for a neural material by illuminating the neural material, initially for a flat plane, and subsequently for an irregular or curved surface, wherein the neural material represents the transferable material; training, using the training data, the neural material by optimizing a value for a loss function comparing the target 3D asset with the neural material for each rendering of the target 3D asset; applying, in response to the optimizing, the neural material as trained to a coarse, geometric proxy of the target 3D asset to encode, using the position features from the multiresolution hash grid, a high-fidelity model of the target 3D asset; and rendering, using the high-fidelity model, a high-fidelity image corresponding to the target 3D asset with the transferable material applied.

2 . The method of claim 1 , further comprising per-face texture mapping an existing digital object to produce the coarse, geometric proxy, wherein the coarse, geometric proxy comprises a polygonal mesh.

3 . The method of claim 1 , wherein rendering the high-fidelity image comprises rendering the high-fidelity model in a virtual scene.

4 . The method of claim 1 , further comprising:
accessing images of a real-world 3D object; and using the images to apply the neural material as trained to the coarse, geometric proxy.

5 . The method of claim 1 , wherein the set of lighting and camera configurations comprises a fixed lighting configuration to provide novel view synthesis.

6 . The method of claim 1 , wherein rendering the high-fidelity image comprises deploying the high-fidelity model to a remote device for the rendering.

7 . The method of claim 1 , wherein training the neural material further comprises using texture coordinates on a surface of the coarse, geometric proxy as input to a neural material function.

8 . A system comprising:
a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising:
accessing a transferable material from a digital asset having an existing geometry;
sampling a set of lighting and camera configurations arranged to render an image of a target 3D asset;
generating position features for the target 3D asset using a multiresolution hash grid;
repeatedly rendering the target 3D asset using the set of lighting and camera configurations;
generating training data for a neural material by illuminating the neural material, initially for a flat plane, and subsequently for an irregular or curved surface, wherein the neural material represents the transferable material;
training, using the training data, the neural material by optimizing a value for a loss function comparing the target 3D asset with the neural material for each rendering of the target 3D asset;
applying, in response to the optimizing, the neural material as trained to a coarse, geometric proxy of the target 3D asset to encode, using the position features from the multiresolution hash grid, a high-fidelity model of the target 3D asset; and
deploying the high-fidelity model to a remote device.

9 . The system of claim 8 , wherein the operations further comprise per-face texture mapping an existing digital object to produce the coarse, geometric proxy, wherein the coarse, geometric proxy comprises a polygonal mesh.

10 . The system of claim 8 , wherein the operations further comprise causing the remote device to render, using the high-fidelity model, a high-fidelity image in a virtual scene.

11 . The system of claim 8 , wherein the operations further comprise:
accessing images of a real-world 3D object; and using the images to apply the neural material as trained to the coarse, geometric proxy.

12 . The system of claim 8 , wherein the set of lighting and camera configurations comprises a fixed lighting configuration to provide novel view synthesis.

13 . The system of claim 8 , wherein the operations further comprise causing the remote device to render a high-fidelity image using the high-fidelity model.

14 . The system of claim 8 , wherein the operation of training the neural material further comprises using texture coordinates on a surface of the coarse, geometric proxy as input to a neural material function.

15 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
accessing a transferable material from a digital asset having an existing geometry; sampling a set of lighting and camera configurations arranged to render an image of a target 3D asset; generating position features for the target 3D asset using a multiresolution hash grid; generating training data for a neural material by illuminating the neural material, initially for a flat plane, and subsequently for an irregular or curved surface, wherein the neural material represents the transferable material; training, using the training data, the neural material by optimizing a value for a loss function comparing the target 3D asset with the neural material for each rendering of the target 3D asset; a step for encoding, in response to the optimizing and using a geometric proxy of the target 3D asset, a high-fidelity model of the target 3D asset based on the set of lighting and camera configurations; and rendering, using the high-fidelity model, a high-fidelity image corresponding to the target 3D asset using the high-fidelity model with the transferable material applied.

16 . The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise per-face texture mapping of an existing digital object to produce a coarse, geometric proxy for use in the step for encoding the high-fidelity model.

17 . The non-transitory computer-readable medium of claim 16 , wherein the geometric proxy comprises a polygonal mesh.

18 . The non-transitory computer-readable medium of claim 15 , wherein the operation of rendering the high-fidelity image further comprises rendering the high-fidelity model in a virtual scene.

19 . The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:
accessing images of a real-world 3D object; and using at least one of the images to encode the high-fidelity model of the target 3D asset.

20 . The non-transitory computer-readable medium of claim 15 , wherein the operation of rendering the high-fidelity image further comprises deploying the high-fidelity model to a remote device for the rendering.

Assignees

• Adobe Inc

Inventors

• Mullia Lakshminarayana Krishna Bhargava
• Deschaintre Valentin
• Carr Nathan
• Hasan Milos
• Miller Bailey

Classifications

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T9/001 Primary

  Model-based coding, e.g. wire frame · CPC title

• G06T15/04

  Texture mapping · CPC title

• G06T15/50 Primary

  Lighting effects · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title