Kernel-based frame interpolation for spatio-temporally adaptive rendering

Patent metadata

Field	Value
Publication number	US-12367544-B2
Application number	US-202318197366-A
Country	US
Kind code	B2
Filing date	May 15, 2023
Priority date	May 18, 2022
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

A system includes a processor and a memory storing software code. The processor executes the software code to receive a first rendered image having first color and auxiliary buffers, receive a second rendered image having second color and auxiliary buffers, provide a third auxiliary buffer for an intermediate image to be interpolated, and identify one or more context features from each of the color buffers and the auxiliary buffers. The software code further interpolates, based on the context features, the intermediate image to provide an interpolated image, determines, based on the one or more context features from each of the first, second, and third auxiliary buffers, a rendering mask for producing one or more rendered sub-portions of the intermediate image, partially renders, using the rendering mask, the intermediate image to provide a partially rendered image, and blends the interpolated image and the partially rendered image to provide a composite image.

First claim

What is claimed is:

1. A system comprising:
a hardware processor; and
a system memory storing a software code;
the hardware processor configured to execute the software code to:
receive a first rendered image having a first color buffer and a first auxiliary buffer;
receive a second rendered image having a second color buffer and a second auxiliary buffer;
provide a third auxiliary buffer for an intermediate image to be interpolated, the intermediate image being between the first rendered image and the second rendered image;
identify one or more context features from each of the first color buffer, the first auxiliary buffer, the second color buffer, the second auxiliary buffer, and the third auxiliary buffer to provide a plurality of context features;
interpolate, based on the plurality of context features, the intermediate image to provide an interpolated image;
determine, based on the identified one or more context features from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer, a rendering mask for producing one or more rendered sub-portions of the intermediate image;
partially render, using the rendering mask, the intermediate image to provide a partially rendered image including the one or more rendered sub-portions; and
blend the interpolated image and the partially rendered image to provide a composite image.

2. The system of claim 1 , wherein in the composite image, the one or more rendered sub-portions of the intermediate image overlay the interpolated image.

3. The system of claim 1 , wherein the rendering mask is determined using a machine learning (ML) model.

4. The system of claim 1 , wherein the one or more context features are identified from each of the first color buffer and the second color buffer using a first ML model, and wherein the one or more context features are identified from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer using a second ML model.

5. The system of claim 1 , wherein determining the rendering mask is not based on the one or more context features identified from either of the first color buffer or the second color buffer.

6. The system of claim 1 , wherein the one or more context features identified from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer comprise one or more of albedo, normal, depth, or velocity.

7. The system of claim 1 , wherein the hardware processor is further configured to execute the software code to:
determine a plurality of kernels for interpolating the intermediate image, wherein the plurality of kernels are determined so as to ensure that the interpolated image comprises a linear combination of the first rendered image and the second rendered image; and
wherein interpolating the intermediate image to provide the interpolated image comprises using the plurality of kernels.

8. The system of claim 1 , wherein the hardware processor is further configured to execute the software code to:
compute a first optical flow from the first rendered image to the intermediate image;
compute a second optical flow from the second rendered image to the intermediate image;
splat, using the first optical flow, the one or more context features identified from each of the first color buffer and the first auxiliary buffer to provide one or more splatted first color features and one or more splatted first auxiliary features; and
splat, using the second optical flow, the one or more context features identified from each of the second color buffer and the second auxiliary buffer to provide one or more splatted second color features and one or more splatted second auxiliary features;
wherein interpolating the intermediate image to provide the interpolated image uses the one or more splatted first color features, the one or more splatted first auxiliary features, the one or more splatted second color features, and the one or more splatted second auxiliary features.

9. The system of claim 8 , wherein determining the rendering mask uses the one or more splatted first auxiliary features and the one or more splatted second auxiliary features, and does not use the one or more splatted first color features or the one or more splatted second color features.

10. The system of claim 8 , wherein computing the first optical flow uses one or more motion vectors of the first rendered image, and wherein computing the second optical flow uses one or more motion vectors of the second rendered image.

11. A method for use by a system including a hardware processor and a system memory storing a software code, the method comprising:
receiving, by the software code executed by the hardware processor, a first rendered image having a first color buffer and a first auxiliary buffer;
receiving, by the software code executed by the hardware processor, a second rendered image having a second color buffer and a second auxiliary buffer;
providing, by the software code executed by the hardware processor, a third auxiliary buffer for an intermediate image to be interpolated, the intermediate image being between the first rendered image and the second rendered image;
identifying, by the software code executed by the hardware processor, one or more context features from each of the first color buffer, the first auxiliary buffer, the second color buffer, the second auxiliary buffer, and the third auxiliary buffer to provide a plurality of context features;
interpolating, by the software code executed by the hardware processor, based on the plurality of context features, the intermediate image to provide an interpolated image;
determining, by the software code executed by the hardware processor, based on the identified one or more context features from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer, a rendering mask for producing one or more rendered sub-portions of the intermediate image;
partially rendering, by the software code executed by the hardware processor and using the rendering mask, the intermediate image to provide a partially rendered image including the one or more rendered sub-portions; and
blending, by the software code executed by the hardware processor, the interpolated image and the partially rendered image to provide a composite image.

12. The method of claim 11 , wherein in the composite image, the one or more rendered sub-portions of the intermediate image overlay the interpolated image.

13. The method of claim 11 , wherein the rendering mask is determined using a machine learning (ML) model.

14. The method of claim 11 , wherein the one or more context features are identified from each of the first color buffer and the second color buffer using a first ML model, and wherein the one or more context features are identified from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer using a second ML model.

15. The method of claim 11 , wherein determining the rendering mask is not based on the one or more context features identified from either of the first color buffer or the second color buffer.

16. The method of claim 11 , wherein the one or more context features identified from each of the first auxiliary buffer, the second auxiliary buffer, and the third auxiliary buffer comprise one or more of albedo, normal, depth, or velocity.

17. The method of claim 11 , further comprising:
determining, by the software code executed by the hardware processor, a plurality of kernels for interpolating the inte

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich
• Eth Zurich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Schroers Christopher Richard
• Briedis Karlis Martins
• Meyer Mark
• Mcgonigal Ian
• Djelouah Abdelaziz
• Otriz Raphael François

Classifications

• G06T2207/10024

  Color image · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title