Neural network system with temporal feedback for denoising of rendered sequences

Patent metadata

Field	Value
Publication number	US-11557022-B2
Application number	US-201916718607-A
Country	US
Kind code	B2
Filing date	Dec 18, 2019
Priority date	Jul 27, 2017
Publication date	Jan 17, 2023
Grant date	Jan 17, 2023

Abstract

A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a sequence of rendered image frames including artifacts, the sequence including a first rendered image frame and a second rendered image frame, wherein each rendered image frame in the sequence of rendered image frames is produced according to a sample map that indicates a number of samples that are computed for each pixel in the rendered image frame;
processing the first rendered image frame using layers of a neural network model to produce external state including a reconstructed first rendered image frame with fewer artifacts compared with the first rendered image frame; and
warping the external state, using difference data corresponding to changes between the first rendered image frame and the second rendered image frame, to produce warped external state, wherein guide data corresponding to the first rendered image frame is processed, based on the warped external state produced for a previous rendered image frame in the sequence, to produce the sample map for the first rendered image frame.

2. The computer-implemented method of claim 1 , wherein the warped external state includes a warped reconstructed first rendered image frame and further comprising processing the second rendered image frame and the warped external state by the neural network model to produce spatially-varying filter kernels.

3. The computer-implemented method of claim 2 , wherein the spatially-varying filter kernels comprises a first filter kernel and a second filter kernel.

4. The computer-implemented method of claim 3 , wherein the second filter kernel is a hierarchy of filter kernels.

5. The computer-implemented method of claim 3 , wherein the processing comprises:
applying the first filter kernel to the reconstructed first rendered image frame to produce a filtered portion of the reconstructed first rendered image frame;
applying the second filter kernel to the second rendered image frame to produce a filtered portion of the second rendered image frame; and
summing the filtered portion of the second rendered image frame and the filtered portion of the reconstructed first rendered image frame to produce a portion of athc reconstructed second rendered image frame.

6. The computer-implemented method of claim 1 , wherein the sample map is generated by an estimator neural network model and further comprising adjusting parameters of the neural network model and the estimator neural network model based on differences between the reconstructed first rendered image frame and a target image frame.

7. The computer-implemented method of claim 1 , wherein the guide data comprises normal vectors, depth, or albedo.

8. The computer-implemented method of claim 1 , wherein the difference data is motion vectors.

9. A system, comprising:
a denoiser neural network model configured to:
receive a sequence of rendered image frames including artifacts, the sequence including a first rendered image frame and a second rendered image frame, wherein each rendered image frame in the sequence of rendered image frames is produced according to a sample map that indicates a number of samples that are computed for each pixel in the rendered image frame; and
process the first rendered image frame using layers of the denoiser neural network model to produce external state including a reconstructed first rendered image frame with fewer artifacts compared with the first rendered image frame; and
a temporal warp function configured to warp the external state, using difference data corresponding to changes between the first rendered image frame and the second rendered image frame, to produce warped external state; and
an estimator neural network model configured to process, based on the warped external state produced for a previous rendered image frame in the sequence, guide data corresponding to the first rendered image frame to produce the sample map for the first rendered image frame.

10. The system of claim 9 , wherein the warped external state includes a warped reconstructed first rendered image frame and the denoiser neural network model is further configured to process the second rendered image frame and the warped external state to produce spatially-varying filter kernels.

11. The system of claim 10 , wherein the spatially-varying filter kernels comprises a first filter kernel and a second filter kernel.

12. The system of claim 11 , wherein the second filter kernel is a hierarchy of filter kernels.

13. The system of claim 11 , wherein the denoiser neural network model is further configured to:
apply the first filter kernel to the reconstructed first rendered image frame to produce a filtered portion of the reconstructed first rendered image frame;
apply the second filter kernel to the second rendered image frame to produce a filtered portion of the second rendered image frame; and
sum the filtered portion of the second rendered image frame and the filtered portion of the reconstructed first rendered image frame to produce a portion of a reconstructed second rendered image frame.

14. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
receive a sequence of rendered image frames including artifacts, the sequence including a first rendered image frame and a second rendered image frame, wherein each rendered image frame in the sequence of rendered image frames is produced according to a sample map that indicates a number of samples that are computed for each pixel in the rendered image frame;
process the first rendered image frame using layers of a neural network model to produce external state including a reconstructed first rendered image frame with fewer artifacts compared with the first rendered image frame;
warp the external state, using difference data corresponding to changes between the first rendered image frame and the second rendered image frame, to produce warped external state, wherein guide data corresponding to the first rendered image frame is processed, based on the warped external state produced for a previous rendered image frame in the sequence, to produce the sample map for the first rendered image frame.

15. The computer-implemented method of claim 1 , further comprising processing the second rendered image frame using the layers of the neural network model to produce a reconstructed second rendered image frame with fewer artifacts compared with the second rendered image frame, wherein the warped external state is input to one or more of the layers of the neural network model.

16. The system of claim 9 , wherein the warped external state is input to one or more of the layers of the denoiser neural network model and the denoiser neural network model processes the second rendered image frame to produce a reconstructed second rendered image frame with fewer artifacts compared with the second rendered image frame.

17. The system of claim 9 , wherein the guide data comprises normal vectors, depth, or albedo.

18. The non-transitory, computer-readable storage medium of claim 14 , wherein the instructions further cause the processing unit to process the second rendered image frame using the layers of the neural network model to produce a reconstructed second rendered image frame with fewer artifacts compared with the second rendered image frame, wherein the warped external state is input to one or more of the layers of the neural network.

19. The non-transitory, computer-readable storage medium of claim 14 ,

Assignees

• Nvidia Corp

Inventors

• Munkberg Carl Jacob
• Hasselgren Jon Niklas Theodor
• Patney Anjul
• Salvi Marco
• Lefohn Aaron Eliot
• Brittain Donald Lee

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title