Learning 2d texture mapping in volumetric neural rendering

Patent metadata

Field	Value
Publication number	US-2024177399-A1
Application number	US-202418426084-A
Country	US
Kind code	A1
Filing date	Jan 29, 2024
Priority date	Dec 23, 2020
Publication date	May 30, 2024
Grant date	—

Abstract

Embodiments are disclosed for neural texture mapping. In some embodiments, a method of neural texture mapping includes obtaining a plurality of images of an object, determining volumetric representation of a scene of the object using a first neural network, mapping 3D points of the scene to a 2D texture space using a second neural network, and determining radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a second neural network to generate a 3D appearance representation of the object.

First claim

We claim:

1 . A computer-implemented method comprising:
receiving a request to edit an appearance of an object, wherein the appearance of the object is generated by a texture mapping network of a volumetric neural rendering system, wherein texture mapping network is trained on a cycle loss using an inverse texture mapping network to enforce a mapping between the appearance of the object and surface points on the object; obtaining a texture map corresponding to the appearance of the object; and editing the texture map based on the request.

2 . The method of claim 1 , wherein receiving a request to edit an appearance of an object, further comprises:
receiving a request to edit the texture map corresponding to the appearance of the object.

3 . The method of claim 1 , wherein editing the texture map based on the request, further comprises:
modifying the texture map based on the request to create a modified texture map.

4 . The method of claim 3 , wherein modifying the texture map based on the request to create a modified texture map includes:
replacing the texture map with a new texture map, wherein the modified texture map is the new texture map.

5 . The method of claim 3 , wherein modifying the texture map based on the request to create a modified texture map includes:
combining the texture map with one or more changes included in the request to edit the appearance of the object to create the new texture map.

6 . The method of claim 1 , wherein the volumetric neural rendering system generates a 3D geometric representation of the object and separately generates a 3D appearance representation of the object.

7 . The method of claim 6 , wherein to generate the 3D geometric representation of the object and separately generate the 3D appearance representation of the object, the volumetric neural rendering system is configured to:
obtain a plurality of images of a scene depicting the object; determine a volume density of the scene using a scene geometry network to generate the 3D geometric representation of the object; map 3D points of the scene to a 2D texture space using the texture mapping network; and determine radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a texture network to generate the 3D appearance representation of the object.

8 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving a request to edit an appearance of an object, wherein the appearance of the object is generated by a texture mapping network of a volumetric neural rendering system, wherein texture mapping network is trained on a cycle loss using an inverse texture mapping network to enforce a mapping between the appearance of the object and surface points on the object; obtaining a texture map corresponding to the appearance of the object; and editing the texture map based on the request.

9 . The non-transitory computer-readable medium of claim 8 , wherein the operation of receiving a request to edit an appearance of an object, further comprises:
receiving a request to edit the texture map corresponding to the appearance of the object.

10 . The non-transitory computer-readable medium of claim 8 , wherein the operation of editing the texture map based on the request, further comprises:
modifying the texture map based on the request to create a modified texture map.

11 . The non-transitory computer-readable medium of claim 10 , wherein the operation of modifying the texture map based on the request to create a modified texture map includes:
replacing the texture map with a new texture map, wherein the modified texture map is the new texture map.

12 . The non-transitory computer-readable medium of claim 10 , wherein the operation of modifying the texture map based on the request to create a modified texture map includes:
combining the texture map with one or more changes included in the request to edit the appearance of the object to create the new texture map.

13 . The non-transitory computer-readable medium of claim 8 , wherein the volumetric neural rendering system generates a 3D geometric representation of the object and separately generates a 3D appearance representation of the object.

14 . The non-transitory computer-readable medium of claim 13 , wherein to generate the 3D geometric representation of the object and separately generate the 3D appearance representation of the object, the volumetric neural rendering system is configured to:
obtain a plurality of images of a scene depicting the object; determine a volume density of the scene using a scene geometry network to generate the 3D geometric representation of the object; map 3D points of the scene to a 2D texture space using the texture mapping network; and determine radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a texture network to generate the 3D appearance representation of the object.

15 . A system comprising:
a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising:
receiving a request to edit an appearance of an object, wherein the appearance of the object is generated by a texture mapping network of a volumetric neural rendering system, wherein texture mapping network is trained on a cycle loss using an inverse texture mapping network to enforce a mapping between the appearance of the object and surface points on the object;
obtaining a texture map corresponding to the appearance of the object; and
editing the texture map based on the request.

16 . The system of claim 15 , wherein the operation of receiving a request to edit an appearance of an object, further comprises:
receiving a request to edit the texture map corresponding to the appearance of the object.

17 . The system of claim 15 , wherein the operation of editing the texture map based on the request, further comprises:
modifying the texture map based on the request to create a modified texture map.

18 . The system of claim 17 , wherein the operation of modifying the texture map based on the request to create a modified texture map includes:
replacing the texture map with a new texture map, wherein the modified texture map is the new texture map.

19 . The system of claim 17 , wherein the operation of modifying the texture map based on the request to create a modified texture map includes:
combining the texture map with one or more changes included in the request to edit the appearance of the object to create the new texture map.

20 . The system of claim 15 , wherein the volumetric neural rendering system generates a 3D geometric representation of the object and separately generates a 3D appearance representation of the object, and wherein to generate the 3D geometric representation of the object and separately generate the 3D appearance representation of the object, the volumetric neural rendering system is configured to:
obtain a plurality of images of a scene depicting the object; determine a volume density of the scene using a scene geometry network to generate the 3D geometric representation of the object; map 3D points of the scene to a 2D texture space using the texture mapping network; and determine radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a texture network to generate

Assignees

• Adobe Inc

Inventors

• Xu Zexiang
• Hold-Geoffroy Yannick
• Hasan Milos
• Sunkavalli Kalyan
• Xiang Fanbo

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title