Deep learning based selection of samples for adaptive supersampling

Patent metadata

Field	Value
Publication number	US-11526964-B2
Application number	US-202016898116-A
Country	US
Kind code	B2
Filing date	Jun 10, 2020
Priority date	Jun 10, 2020
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

An apparatus to facilitate deep learning based selection of samples for adaptive supersampling is disclosed. The apparatus includes one or more processing elements to: receive training data comprising input tiles and corresponding supersampling values for the input tiles, wherein each input tile comprises a plurality of pixels, and train, based on the training data, a machine learning model to identify a level of supersampling for a rendered tile of pixels.

First claim

What is claimed is:

1. An apparatus comprising:
one or more processing elements to:
receive training data comprising:
a first rendered version of an input tile of an image, the first rendered version of the input tile rendered using a first supersampling value applied to a plurality of pixels of the input tile, wherein the first supersampling value comprises a first number of times to sample the plurality of pixels of input tile during a first render process for the image; and
a second supersampling value for the input tile, wherein the second supersampling value for the input tile comprises a second number of times to sample the plurality of pixels of the input tile during a second render process for the image to cause a second rendered version of the input tile to have a quality measurement metric value that satisfies a determined quality measurement metric threshold value; and

train, based on the training data, a machine learning model to identify the second supersampling value for the first rendered version of the input tile, wherein a trained version of the machine learning model is to predict a proposed supersampling value for a rendered tile that is input to the trained version of the machine learning model to avoid oversampling of the rendered tile.

2. The apparatus of claim 1 , wherein the input tile comprises at least one of 8×8 pixels, 16×16 pixels, or 32×32 pixels of a rendered image.

3. The apparatus of claim 1 , wherein the one or more processing elements are comprised in a graphics processing unit (GPU).

4. The apparatus of claim 1 , wherein the first supersampling value is set to 1.

5. The apparatus of claim 1 , wherein the quality measurement metric value comprising a Structural Similarity Index measurement (SSIM) value.

6. The apparatus of claim 1 , wherein the quality measurement metric value comprising a Peak Signal to Noise Ratio (PSNR) measurement value.

7. The apparatus of claim 1 , wherein the one or more processing elements to train the machine learning model using an offline process that is separate from a real-time usage of the machine learning model during an inference phase.

8. The apparatus of claim 1 , wherein the training data further comprises depth values corresponding to the input tile.

9. The apparatus of claim 1 , wherein the training data further comprises at least one of normal, object IDs, texture colors, primitive IDs, or temporal data corresponding to a temporally-previous rendered version of the image.

10. The apparatus of claim 1 , wherein the machine learning model is trained using a convolutional neural network (CNN).

11. The apparatus of claim 10 , wherein the CNN comprises at least one of an input layer, one or more convolutional layers, at least one flatten layer, and one or more dense functions, and wherein the CNN utilizes an Adaptive Moment Estimation (ADAM) optimizer and a mean square error (MSE) loss function.

12. The apparatus of claim 1 , wherein the machine learning model as trained by the one or more processing elements is applied to rendered tiles of pixels for at least one of rasterization, ray tracing, variable rate shading (VRS), coarse pixel shading (CPS), hybrid rendering, virtual reality (VR), or augmented reality (AR).

13. The apparatus of claim 1 , wherein the first supersampling value and the second supersampling value are a same value.

14. A method comprising:
rendering a tile without applying supersampling, the tile comprising a plurality of pixels;
providing the rendered tile as input to a trained machine learning model, wherein the trained machine learning model is trained based on training data comprising at least a training input tile rendered with a first supersampling value and a corresponding second supersampling value for the training input tile, and wherein the corresponding second supersampling value is determined to cause another rendered version of the training input tile using the second supersampling value to have a quality measurement metric value that exceeds a quality measurement metric threshold value;
receiving, from the trained machine learning model, a proposed supersampling value for the rendered tile, wherein the proposed supersampling value comprises a number of times to sample pixels of the rendered tile; and
re-rendering the tile with supersampling using the proposed supersampling value received from the trained machine learning model.

15. The method of claim 14 , wherein the proposed supersampling value is provided in a power of two format.

16. The method of claim 14 , wherein the first supersampling value is set to 1.

17. The method of claim 14 , wherein a smoothing function is applied to rendered tiles of pixels having a difference between supersampling levels that exceeds a determined threshold, the supersampling levels provided by the trained machine learning model.

18. A non-transitory computer-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to:
receive training data comprising:
a first rendered version of an input tile of an image, the first rendered version of the input tile rendered using a first supersampling value applied to a plurality of pixels of the input tile, wherein the first supersampling value comprises a first number of times to sample the plurality of pixels of input tile during a first render process for the image; and
a second supersampling value for the input tile, wherein the second supersampling value for the input tile comprises a second number of times to sample the plurality of pixels of the input tile during a second render process for the image to cause a second rendered version of the input tile to have a quality measurement metric value that satisfies a determined quality measurement metric threshold value; and

train, based on the training data, a machine learning model to identify the second supersampling value for the first rendered version of the input tile, wherein a trained version of the machine learning model is to predict a proposed supersampling value for a rendered tile that is input to the trained version of the machine learning model to avoid oversampling of the rendered tile.

19. The non-transitory computer-readable medium of claim 18 , wherein the first supersampling value is set to 1.

20. The non-transitory computer-readable medium of claim 18 , wherein the quality measurement metric value comprising at least one of a Structural Similarity Index measurement (SSIM) measurement value or a Peak Signal to Noise Ratio (PSNR) measurement value.

Assignees

• Intel Corp

Inventors

• Pohl Daniel
• Marshall Carl
• Panneer Selvakumar

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T1/60

  Memory management · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title