Hardware-efficient neural frame prediction with low resolution optical flow

Patent metadata

Field	Value
Publication number	US-2024311962-A1
Application number	US-202318308725-A
Country	US
Kind code	A1
Filing date	Apr 28, 2023
Priority date	Mar 16, 2023
Publication date	Sep 19, 2024
Grant date	—

Abstract

Described herein are techniques to enhance the user experience for 3D rendered applications via neural frame generation using upsampled optical flow data. In one embodiment, a neural network is trained using both sparse optical flow data and dense optical flow data to enable neural frame generation to be performed by a deployed neural network using only sparse optical flow data. The sparse optical flow data can be upsampled to dense optical flow data by the trained neural network. The neural network can then use the upsampled dense optical flow data to perform frame generation.

First claim

What is claimed is:

1 . A data processing system comprising:
a memory device configured to store instructions; a parallel processor including circuitry configured to perform matrix operations, the parallel processor configured to:
load extrapolation or interpolation weights associated with a machine learning model that is configurable to estimate an optical flow via one of extrapolation and interpolation;
perform operations associated with the machine learning model, the operations to estimate a predicted optical flow based on loaded weights, a plurality of rendered frames at a first resolution, and an optical flow between the plurality of rendered frames at the first resolution, the optical flow upsampled to the first resolution from input optical flow at a second resolution that is less than the first resolution; and
generate a predicted frame based on a rendered frame of the plurality of rendered frames and the predicted optical flow via one of extrapolation and interpolation.

2 . The data processing system of claim 1 , the parallel processor configured to warp a rendered frame of the plurality of rendered frames based on the predicted optical flow to generate the predicted frame.

3 . The data processing system of claim 2 , the parallel processor configured to:
upsample the input optical flow from the second resolution to the first resolution via operations associated with an optical flow upsampling and denoising subnetwork of the machine learning model; and denoise upsampled optical flow via a denoising filter predicted by operations performed for the optical flow upsampling and denoising subnetwork.

4 . The data processing system of claim 3 , the parallel processor configured to predict the denoising filter based on the input optical flow at the second resolution.

5 . The data processing system of claim 4 , the parallel processor configured to:
receive red, green, and blue (RGB) color data for the plurality of rendered frames; receive optical flow between the plurality of rendered frames, the optical flow including upsampled and denoised optical flow; preprocess the RGB color data for the plurality of rendered frames to generate preprocessed RGB data; concatenate the preprocessed RGB data and the optical flow between the plurality of rendered frames into a multi-channel block of input data; and generate the predicted optical flow based on the multi-channel block of input data via operations associated with the machine learning model and the loaded weights.

6 . The data processing system of claim 5 , wherein to preprocess the RGB color data, the parallel processor is configured to:
calculate an overall mean for the plurality of rendered frames; and subtract the overall mean from each frame of the plurality of rendered frames.

7 . The data processing system of claim 5 , the parallel processor configured to upsample and denoise the input optical flow and generate the predicted optical flow via the circuitry configured to perform matrix operations.

8 . The data processing system of claim 7 , wherein to upsample and denoise the input optical flow, the parallel processor is configured to:
load optical flow data that includes the input optical flow at the second resolution into the circuitry configured to perform matrix operations; predict a denoising filter based on the optical flow data; upsample the optical flow data according to an upsample factor to generate intermediate optical flow data at the first resolution; and apply a predicted denoising filter to the intermediate optical flow data to generate upsampled and denoised optical flow data at the first resolution.

9 . The data processing system of claim 7 , wherein to generate the predicted optical flow, the parallel processor is configured to:
load the multi-channel block of input data into the circuitry configured to perform matrix operations; and process the multi-channel block of input data via matrix operations associated with the machine learning model according to a channel dimension parameter and a filter size.

10 . The data processing system of claim 9 , wherein to process the multi-channel block of input data, the parallel processor is configured to:
process the multi-channel block of input data according to the channel dimension parameter via operations associated with an encoder portion of the machine learning model, the encoder portion including a first plurality of convolution layers; process output of the encoder portion according to the channel dimension parameter via operations associated with a decoder portion of the machine learning model, the decoder portion including a plurality of transposed convolution layers and a second plurality of convolution layers; and process output of the decoder portion according to the filter size parameter via operations associated with a denoising portion of the machine learning model, the decoder portion to output the predicted optical flow or the interpolated optical flow.

11 . The data processing system of claim 10 , wherein to process output of the encoder portion, the parallel processor is configured to:
generate intermediate output associated with a first transposed convolution layer of the plurality of transposed convolution layers; perform operations associated with a first convolution layer of the second plurality of convolution layers on the intermediate output to estimate an intermediate optical flow; and concatenate the intermediate optical flow with the intermediate output and associated decoder stage output received via a skip connection between the decoder portion and the encoder portion to generate input for a second transposed convolution layer of the plurality of transposed convolution layers.

12 . A method comprising:
performing end-to-end training of a neural frame prediction network with an extrapolation dataset to generate extrapolation weights for frame generation and optical flow upsampling; performing end-to-end training of the neural frame prediction network with an interpolation dataset to generate interpolation weights for frame generation and optical flow upsampling; and deploying extrapolation weights and interpolation weights along with the neural frame prediction network, the neural frame prediction network to enable a graphics processor to perform operations comprising:
upsampling optical flow data to a first resolution from input optical flow at a second resolution that is less than a first resolution,
denoising optical flow data upsampled to the first resolution, and
generating a frame via one of interpolation and extrapolation via upsampled optical flow data at the first resolution.

13 . The method of claim 12 , wherein performing end-to-end training of the neural frame prediction network with the extrapolation or the interpolation dataset includes:
preprocessing red, green, and blue (RGB) data associated with a plurality of rendered frames, the plurality of rendered frames at the first resolution; performing a forward pass for a first sub-network of the neural frame prediction network, the first sub-network to upsample the optical flow data from the second resolution to the first resolution and denoise upsampled optical flow data; and performing a forward pass of a second sub-network of the neural frame prediction network, the second sub-network to predict an optical flow between a rendered frame and a generated frame based on the plurality of rendered frames at the first resolution and upsampled and denoised optical flow data generated by the first sub-network.

14 . The method of claim 13 , further comprising:
genera

Assignees

• Intel Corp

Inventors

• Iyer Darshan R
• Vembar Deepak
• Wang Changliang
• Bhatia Sumit

Classifications

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06T2207/10024

  Color image · CPC title