Rendering an image of a 3-D scene using guided image filtering

Patent metadata

Field	Value
Publication number	US-12198307-B2
Application number	US-202217956778-A
Country	US
Kind code	B2
Filing date	Sep 29, 2022
Priority date	Sep 30, 2021
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

A method of rendering an image of a 3-D scene includes rendering a noisy image at a first resolution; obtaining one or more guide channels at the first resolution, and obtaining one or more corresponding guide channels at a second resolution. The second resolution may be the same resolution as, or a higher resolution than, the first resolution. For each of a plurality of local neighbourhoods, the method comprises: calculating the parameters of a model that approximates the noisy image as a function of the one or more guide channels (at the first resolution), and applying the calculated parameters to the one or more guide channels at the second resolution, to produce a denoised image at the second resolution.

First claim

What is claimed is:

1. A method of rendering an image of a 3-D scene, the method comprising:
rendering a noisy low-resolution image;
obtaining one or more low-resolution guide channels and obtaining one or more corresponding full-resolution guide channels; and
for each of a plurality of local neighbourhoods of the low resolution image:
calculating the parameters of a model that approximates the noisy low-resolution image as a function of the one or more low-resolution guide channels, and
applying the calculated parameters to the one or more full-resolution guide channels, to produce a full-resolution denoised image.

2. The method of claim 1 , wherein rendering the noisy low-resolution image comprises rendering by path tracing.

3. The method of claim 1 , wherein the noisy low-resolution image comprises (i) indirect lighting in the scene or (ii) an ambient occlusion image.

4. The method of claim 3 , wherein the full-resolution denoised image is a full-resolution denoised indirect lighting image or a full-resolution denoised ambient occlusion image, and wherein the method further comprises:
obtaining a full-resolution direct lighting image; and
combining the full-resolution denoised image with the full-resolution direct lighting image to produce a full-resolution global illumination image.

5. The method of claim 4 , wherein obtaining the full-resolution direct lighting image comprises rendering it by ray-tracing or rendering it by rasterization.

6. The method of claim 3 , wherein the noisy low-resolution image is a noisy low-resolution global illumination image, comprising direct and indirect lighting in the scene, whereby the full-resolution denoised image is a full-resolution denoised global illumination image.

7. The method of claim 4 , further comprising combining the full-resolution global illumination image or the full-resolution denoised global illumination image with a surface reflectance image to produce a rendered image of the 3-D scene.

8. The method of claim 1 , wherein obtaining the one or more low-resolution guide channels and/or obtaining the one or more full-resolution guide channels comprises rendering by rasterization.

9. The method of claim 1 , comprising:
defining a first tile, defining respective first contiguous portions of the noisy low-resolution image and the one or more low-resolution guide channels, each comprising a first plurality of pixels;
defining a second tile, defining respective second contiguous portions of the noisy low-resolution image and the one or more low-resolution guide channels, each comprising a second plurality of pixels;
calculating a first outer product between each pixel in the one or more low-resolution guide channels and itself; and
calculating a second outer product between each pixel in the one or more low-resolution guide channels and the corresponding pixel in the noisy low-resolution image,
wherein the first outer product and second outer product are calculated for pixels in the first tile either (i) before the second tile or (ii) concurrently with the second tile.

10. The method of claim 1 , wherein at least one of the noisy low-resolution image, the one or more low-resolution guide channels, the one or more corresponding full-resolution guide channels, and the denoised image are stored in a quantized low-bitdepth format.

11. The method of claim 10 , further comprising, after rendering the noisy low-resolution image, quantizing it in a quantized low-bitdepth format with nonlinear quantization, such that darker regions of the image are quantized to a relatively greater density of quantization levels, and lighter regions of the image are quantized to a relatively lesser density of quantization levels, and storing the quantized low-bitdepth format in a memory,
wherein the method further comprises, before calculating the parameters of the model, retrieving the quantized low-bitdepth value from the memory and performing inverse quantization.

12. The method of claim 1 , wherein calculating the parameters of the model comprises:
calculating a first outer product between each pixel in the one or more low-resolution guide channels and itself;
calculating a second outer product between each pixel in the one or more low-resolution guide channels and the corresponding pixel in the noisy low-resolution image;
blurring the first outer products to calculate a first moment matrix for each local neighbourhood;
blurring the second outer products to calculate a second moment matrix for each local neighbourhood; and
calculating the parameters of the model for each local neighbourhood, comprising calculating an inverse matrix of the first moment matrix, and calculating a product of the inverse matrix and the second moment matrix.

13. The method of claim 12 , wherein blurring the first outer products comprises calculating a first multiscale pyramid from the first outer products and calculating the first moment matrix based on the first multiscale pyramid; and/or
wherein blurring the second outer products comprises calculating a second multiscale pyramid from the second outer products and calculating the second moment matrix based on the second multiscale pyramid.

14. The method of claim 12 , wherein the blurring comprises separable filtering in horizontal and vertical directions.

15. The method of claim 12 , wherein the blurring comprises filtering using an anisotropic 2-D filter.

16. The method of claim 13 , wherein the one or more low-resolution guide channels include surface normals of objects in the 3-D scene, and wherein the blurring comprises:
for each local neighbourhood,
determining a major axis and minor axis of a 2-D filter, based on the surface normal of the object at the centre of the neighbourhood;
selecting a level of the multiscale pyramid, based on the length of the minor axis; and
sampling the selected level of the multiscale pyramid along the major axis.

17. The method of claim 12 , wherein the blurring comprises one of:
IIR filtering; and
filtering with a running box filter.

18. The method of claim 12 , comprising:
defining a first outer product tile, defining a first contiguous portion of the first outer product and a respective first contiguous portion of the second outer product, each comprising a first plurality of pixels; and
defining a second outer product tile, defining a second contiguous portion of the first outer product and a respective second contiguous portion of the second outer product, each comprising a second plurality of pixels,
wherein the first moment matrix and second moment matrix are calculated for the first tile either (i) before the second tile or (ii) concurrently with the second tile.

19. The method of claim 12 , further comprising normalizing and/or regularizing one or both of the first moment matrix and the second moment matrix.

20. A non-transitory computer readable storage medium having stored thereon computer readable code configured to cause a method as set forth in claim 1 to be performed when the code is run.

Assignees

• Imagination Tech Ltd

Inventors

• Cséfalvay Szabolcs
• Imber James
• Walton David
• Yu Insu

Classifications

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T15/506

  Illumination models · CPC title