4d generative models from in the wild videos

Patent metadata

Field	Value
Publication number	US-2026099987-A1
Application number	US-202519352319-A
Country	US
Kind code	A1
Filing date	Oct 7, 2025
Priority date	Oct 9, 2024
Publication date	Apr 9, 2026
Grant date	—

Abstract

At least one embodiment for generating 4D generative models from in the wild videos includes receiving a first 2D image frame, processing the first 2D image frame to generate 3D Gaussians, defining a set of motion basis features from the 3D Gaussians, receiving a second 2D image frame, generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame, processing the plurality of augmented images to generate a plurality of motion features, constructing deformed 3D Gaussians from the motion basis features and the motion features, generating a rendered 2D image from the deformed 3D Gaussians, and generating a 4D representation using a neural network trained based on the rendered 2D image and the second 2D image frame.

First claim

What is claimed is:

1 . A computer-implemented method for generating four-dimensional (4D) representation, the method comprising:
receiving a first two-dimensional (2D) image frame; processing the first 2D image frame to generate a plurality of 3D Gaussians; defining a set of motion basis features from the plurality of 3D Gaussians; receiving a second 2D image frame; generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame; processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features; constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features; generating a rendered 2D image from the plurality of deformed 3D Gaussians; and generating a 4D representation using a trained neural network.

2 . The computer-implemented method of claim 1 , further comprising transforming the first 2D image frame into a 3D representation by aligning features of the first 2D image frame along a plurality of orthogonal planes.

3 . The computer-implemented method of claim 2 , further comprising generating a plurality of feature vectors by sampling a plurality of 3D points along rays and projecting each of the plurality of 3D points onto the plurality of orthogonal planes.

4 . The computer-implemented method of claim 1 , wherein constructing the plurality of deformed 3D Gaussians comprises translating the plurality of 3D Gaussians based on the plurality of motion features.

5 . The computer-implemented method of claim 1 , wherein the trained neural network is trained by minimizing rendering loss between the rendered 2D image and the second 2D image frame.

6 . The computer-implemented method of claim 1 , wherein generating the motion features comprises processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images using a vision transformer.

7 . The computer-implemented method of claim 1 , wherein generating the rendered 2D image comprises performing splatting using the plurality of deformed 3D Gaussians.

8 . The computer-implemented method of claim 1 , further comprising generating a 4D video using the 4D representation.

9 . The computer-implemented method of claim 8 , wherein generating the 4D video comprises:
receiving an audio input; concatenating the audio input and noise removing noise from the concatenated audio input and noise to generate denoised audio features; and generating the 4D video based on the denoised audio features and the 4D representation.

10 . The computer-implemented method of claim 9 , wherein generating the 4D video comprises processing the 4D representation with a diffusion model.

11 . The computer-implemented method of claim 10 , wherein the diffusion model is trained by:
receiving an audio input; generating noisy features by generating a sequence where, at each step, noise is added to the audio input; and generating predicted denoised features by generating a sequence, where, at each step noise is iteratively removed.

12 . The computer-implemented method of claim 10 , wherein the diffusion model is trained by diffusion forcing.

13 . The computer-implemented method of claim 10 , wherein the noise comprises Gaussian noise.

14 . One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
receiving a first 2D image frame; processing the first 2D image frame to generate a plurality of 3D Gaussians; defining a set of motion basis features from the plurality of 3D Gaussians; receiving a second 2D image frame; generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame; processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features; constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features; generating a rendered 2D image from the plurality of deformed 3D Gaussians; generating a 4D representation using a trained neural network.

15 . The one or more non-transitory computer-readable media of claim 14 , wherein the steps further comprise transforming the first 2D image frame into a 3D representation by aligning the features of the first 2D image frame along a plurality of orthogonal planes.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein the steps further comprise generating a plurality of feature vectors by sampling a plurality of 3D points along rays and projecting each of the plurality of 3D points onto the plurality of orthogonal planes.

17 . The one or more non-transitory computer-readable media of claim 14 , wherein constructing the plurality of deformed 3D Gaussians comprises translating the plurality of 3D Gaussians based on the plurality of motion features.

18 . The one or more non-transitory computer-readable media of claim 14 , wherein training the trained neural network comprises minimizing rendering loss between the rendered 2D image and the second 2D image frame.

19 . The one or more non-transitory computer-readable media of claim 14 , further comprising generating a 4D video using the 4D representation.

20 . A system, comprising:
one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform steps comprising:
receiving a first 2D image frame;
processing the first 2D image frame to generate a plurality of 3D Gaussians;
defining a set of motion basis features from the plurality of 3D Gaussians;
receiving a second 2D image frame;
generating a plurality of augmented images based on the first 2D image frame and the second 2D image frame;
processing the first 2D image frame, the second 2D image frame, and the plurality of augmented images to generate a plurality of motion features;
constructing a plurality of deformed 3D Gaussians from the motion basis features and the motion features;
generating a rendered 2D image from the plurality of deformed 3D Gaussians;
generating a 4D representation using a trained neural network.

Assignees

• Nvidia Corp

Inventors

• Nagano Koki
• Jiang Kaiwen
• De Mello Shalini
• Li Xueting

Classifications

• H04N21/816

  involving special video data, e.g 3D video · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T15/08 Primary

  Volume rendering · CPC title