Denoising Monte Carlo renderings using progressive neural networks

Patent metadata

Field	Value
Publication number	US-11037274-B2
Application number	US-202016789025-A
Country	US
Kind code	B2
Filing date	Feb 12, 2020
Priority date	Apr 6, 2017
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.

First claim

What is claimed is:

1. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising:
receiving a first set of input images rendered by MC path tracing, and a first set of corresponding reference images;
receiving a second set of input images rendered by MC path tracing, and a second set of corresponding reference images;
configuring a first neural network including a first plurality of layers and a first number of nodes associated with a first number of parameters, the first neural network configured to receive the first set of input images;
configuring a second neural network including a second plurality of layers and a second number of nodes associated with a second number of parameters, the second neural network configured to receive the second set of input images;
configuring a third neural network including a third plurality of layers and a third number of nodes associated with a third number of parameters, the third neural network configured to:
receive output from one of the first neural network and the second neural network; and
output an output image corresponding to a respective input image;

training the first neural network, the second neural network, and the third neural network to obtain a first number of optimized parameters associated with the first number of nodes of the first neural network, a second number of optimized parameters associated with the second number of nodes of the second neural network, and a third number of optimized parameters associated with the third number of nodes of the third neural network, wherein the training uses the first set of input images and the first set of reference images while the third neural network receives output from the first neural network, and uses the second set of input images and the second set of reference images while the third neural network receives output from the second neural network;
receiving a third set of input images rendered by MC path tracing and a third set of corresponding reference images;
configuring a fourth neural network including a fourth plurality of layers and a fourth number of nodes associated with a fourth number of parameters, the fourth neural network configured to receive the third set of input images, wherein the third neural network is further configured to receive output from the fourth neural network; and
training the fourth neural network using the third set of input images and the third set of reference images to obtain a fourth number of optimized parameters associated with the fourth number of nodes of the fourth neural network, while the third neural network is configured to receive output from the fourth neural network and the third number of optimized parameters associated with the third number of nodes of the third neural network are fixed.

2. The computer product of claim 1 , wherein the instructions further comprise:
receiving a new input image rendered by MC path tracing; and
generating a denoised image corresponding to the new input image by passing the new input image through the fourth neural network and the third neural network using the third number of optimized parameters of the third neural network and the fourth number of optimized parameters of the fourth neural network.

3. The computer product of claim 1 , wherein each of the first neural network and the second neural network comprises a multilayer perceptron neural network.

4. The computer product of claim 1 , wherein each of the first neural network and the second neural network comprises a convolutional neural network.

5. The computer product of claim 1 , wherein the third neural network comprises a reconstruction module for reconstructing a respective output image corresponding to a respective input image.

6. The computer product of claim 5 , wherein the third neural network further comprises a kernel prediction module coupled to the reconstruction module, the kernel prediction module configured to generate a plurality of weights associated with a neighborhood of pixels around each pixel of a respective input image, wherein the reconstruction module is configured to reconstruct the respective output image using the plurality of weights.

7. The computer product of claim 6 , wherein the plurality of weights is normalized.

8. The computer product of claim 1 , wherein each of the first neural network and the third neural network includes less number of layers than the second neural network.

9. The computer product of claim 1 , wherein the first set of input images is rendered by a first renderer of a first type, the second set of input images is rendered by a second renderer of a second type different from the first type, and the third set of input images is rendered by a third renderer of third type different from the first type and the second type.

10. The computer product of claim 1 , wherein the first set of input images includes a first type of image content, the second set of input images includes a second type of image content different from the first type of image content, and the third set of input images includes a third type of image content different from the first type and the second type of image content.

11. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising:
receiving a first set of input images rendered by MC path tracing, and a first set of corresponding reference images;
configuring a first neural network including a first plurality of layers and a first number of nodes associated with a first number of parameters, the first neural network configured to receive the first set of input images;
configuring a second neural network including a second plurality of layers and a second number of nodes associated with a second number of parameters, the second neural network configured to receive output from the first neural network and output an output image corresponding to a respective input image;
training the first neural network and the second neural network using the first set of input images and the first set of reference images to obtain a first number of optimized parameters associated with the first number of nodes of the first neural network and a second number of optimized parameters associated with the second number of nodes of the second neural network;
receiving a second set of input images rendered by MC path tracing and a second set of corresponding reference images;
configuring a third neural network including a third plurality of layers and a third number of nodes associated with a third number of parameters, the third neural network configured to receive the second set of input images, wherein the second neural network is further configured to receive output from the third neural network; and
training the third neural network using the second set of input images and the second set of reference images to obtain a third number of optimized parameters associated with the third number of nodes of the third neural network, while the second neural network is configured to receive output from the third neural network and the second number of optimized parameters associated with the second number of nodes of the second neural network are fixed.

12. The computer product of claim 11 , wherein the instructions further comprise:
receiving a new input image rendered by MC path tracing; and
generating a denoised ima

Assignees

• Pixar
• Disney Entpr Inc

Inventors

• Vogels Thijs
• Rousselle Fabrice
• Mcwilliams Brian
• Meyer Mark
• Novak Jan

Classifications

• G06V10/7747

  Organisation of the process, e.g. bagging or boosting · CPC title

• G06V10/72

  Data preparation, e.g. statistical preprocessing of image or video features · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title