Methods and systems for generating enhanced light texture data

Patent metadata

Field	Value
Publication number	US-2024320903-A1
Application number	US-202318309283-A
Country	US
Kind code	A1
Filing date	Apr 28, 2023
Priority date	Mar 21, 2023
Publication date	Sep 26, 2024
Grant date	—

Abstract

System, method and media for processing a first light data structure that specifies, for each of plurality of directions for each of a plurality surface regions corresponding to a scene, respective light measurements, including: applying a trained artificial intelligence (AI) model to the first light texture data structure to generate an enhanced light data structure that specifies, for each of the plurality of directions for each of the plurality surface regions corresponding to the scene, respective enhanced light measurements; and storing an enhanced scene model that includes the enhanced light data structure together with geometric data that maps the enhanced light measurements.

First claim

1 . A computer implemented method for processing a first light data structure that includes light texture data that specifies, for each of plurality of directions for each of a plurality surface regions corresponding to a scene, respective light measurements, comprising:
applying a trained artificial intelligence (AI) model to the first light data structure to generate an enhanced light data structure that specifies, for each of the plurality of directions for each of the plurality surface regions corresponding to the scene, respective enhanced light measurements; and storing an enhanced scene model that includes the enhanced light data structure together with geometric data that maps the enhanced light measurements within the enhanced light data structure to the plurality surface regions.

2 . The method of claim 1 wherein:
the first light data structure includes visibility probability data that specifies, for each of the plurality of surface regions corresponding to the scene, respective visibility probability values.

3 . The method of claim 2 wherein:
the first light data structure comprises, for each of the plurality of directions, a respective direction specific first light texture map specifying the respective light measurements for the plurality of surface regions for the direction;
the visibility probability data is arranged as a visibility map having the same resolution as the first light texture maps;
and
the enhanced light texture data structure comprises, for each of the plurality of directions, a respective direction specific enhanced light texture map specifying the respective enhanced light measurements for the plurality of surface regions for the direction.

4 . The method of claim 3 wherein the direction specific first light texture are each formatted as respective two dimensional image files, and the geometric data maps respective pixel locations within the two dimensional image files to respective surface regions of the plurality of surface regions.

5 . The method of claim 4 wherein the AI model comprises at least one of a convolutional auto encoder, a vision transformer, or a recurrent neural network.

6 . The method of claim 1 wherein the AI model comprises at least one or more of: a denoiser, a super-sampler; or an anti-aliasing model.

7 . The method of claim 1 wherein:
the first light data structure comprises a respective surface region light texture tensor for each of the respective surface regions, each surface region light texture tensor specifying the respective light measurements for the surface region for the plurality of directions; and
the enhanced light texture data structure comprises a respective enhanced surface region light texture tensor for each of the respective surface regions, each enhanced surface light texture region tensor specifying the respective enhanced light measurements for the surface region for the plurality of directions.

8 . The method of claim 1 wherein the respective light measurements each represent a gathered RGB color value light color measurement.

9 . The method of claim 1 wherein the enhanced scene model conforms to a graphics language transmission format (gITF).

10 . The method of claim 1 wherein, for each of the plurality of directions for each of the plurality surface regions, the light measurements represent light data for a respective range of light directions that intersect the surface region.

11 . The method of claim 1 comprising generating the first light data structure, including:
defining, for each of the plurality of surface regions, a respective local reference frame and a bin structure, the bin structure discretizing the local reference frame into a set of bins, each bin corresponding to a respective range of light directions that intersect the surface region;
computing, for each surface region, a respective color measurement for each bin of the bin structure, the respective color measurement for each bin being based on a path trace of one or more light ray samples that fall within the respective range of light directions corresponding to the bin, wherein the respective color measurements are used as the respective light measurements;
assembling the first light data structure that indicates the respective local reference frames, bin structures, and respective light measurements for the surface regions; and
storing the first light data structure.

12 . The method of claim 1 comprising sending the enhanced scene model through a network to a remote rendering device.

13 . The method of claim 12 comprising repeatedly performing the applying, storing and sending in order to support real-time rendering of series of scenes at the rendering device wherein the applying, storying, and sending are performed at a cloud computing platform that is more computationally powerful than the rendering device.

14 . The method of claim 13 comprising:
at the rendering device:
obtaining the enhanced scene model;
rendering a scene image for the scene model based on an input view direction, wherein pixel colors in the rendered scene image are determined based on the light measurements included in the enhanced scene model.

15 . A system comprising one or more processors and one or more non-transitory memories that store executable instructions for the one or more processors, wherein the executable instructions, when executed by the one or more processors, configure the system to perform a method of processing a first light data structure that includes light texture data that specifies, for each of plurality of directions for each of a plurality surface regions corresponding to a scene, respective light measurements, comprising:
applying a trained artificial intelligence (AI) model to the first light data structure to generate an enhanced light data structure that specifies, for each of the plurality of directions for each of the plurality surface regions corresponding to the scene, respective enhanced light measurements; and storing an enhanced scene model that includes the enhanced light data structure together with geometric data that maps the enhanced light measurements within the enhanced light data structure to the plurality surface regions.

16 . The system of claim 15 wherein:
the first light texture data structure includes visibility probability data that specifies, for each of the plurality surface regions corresponding to the scene, respective visibility probability values.

17 . The system of claim 16 wherein:
the first light data structure comprises, for each of the plurality of directions, a respective direction specific first light texture map specifying the respective light measurements for the plurality of surface regions for the direction;
the visibility probability data is arranged as a visibility map having the same resolution as the first light texture maps; and
the enhanced light texture data structure comprises, for each of the plurality of directions, a respective direction specific enhanced light texture map specifying the respective enhanced light measurements for the plurality of surface regions for the direction.

18 . The system of claim 17 wherein the direction specific first light texture are each formatted as respective two dimensional image files, and the geometric data maps respective pixel locations within the two dimensional image files to respective surface regions of the plurality of surface regions.

19 . The system of claim 14 wherein the method comprises sending the enhanced scene model through a

Assignees

• Liu Yang
• Liu Keyi
• Ibrahim Mohamed
• Sachkov Sergei

Inventors

• Liu Yang
• Liu Keyi
• Ibrahim Mohamed
• Sachkov Sergei

Classifications

• G06T15/04

  Texture mapping · CPC title

• G06T15/06

  Ray-tracing · CPC title

• G06T15/506 Primary

  Illumination models · CPC title