General data prior based on langevin diffusion

Patent metadata

Field	Value
Publication number	US-2025259277-A1
Application number	US-202418884344-A
Country	US
Kind code	A1
Filing date	Sep 13, 2024
Priority date	Feb 12, 2024
Publication date	Aug 14, 2025
Grant date	—

Abstract

Embodiments of the present disclosure relate to a general image prior based on Langevin diffusion. An original representation (image, 3D model, audio) is optimized to resemble a data distribution learned by a trained diffusion model. The original representation may be incomplete and is completed by the optimization process. In an embodiment, the diffusion model may be trained to use an additional conditioning input, such as a text prompt. The diffusion model receives a noisy latent as input and generates a denoised output, such as an image. Generally, the diffusion model samples the learned data distribution to produce the output. The conditioning input provides additional constraint that causes the output to be “nudged” towards the learned data distribution. An example synthesis problem is to generate a panorama image that is much larger compared with images used to train the diffusion model.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
incorporating neutral noise into a representation to produce a latent representation; applying a Langevin diffusion operation to the latent representation to modify the neutral noise based on a desired data distribution, producing a modified latent representation; extracting a difference between a denoised latent produced by denoising the latent representation and a denoised modified latent produced by denoising the modified latent representation; and updating the representation based on the difference.

2 . The computer-implemented method of claim 1 , wherein the representation is obtained by rendering an original representation, and further comprising updating the original representation based on the difference.

3 . The computer-implemented method of claim 2 , wherein the original representation is associated with a first domain and the representation is associated with a second domain.

4 . The computer-implemented method of claim 2 wherein the original representation comprises at least one of an extended video, a panorama image, a 3D model, or extended audio data.

5 . The computer-implemented method of claim 2 , wherein the original representation is defined by a neural radiance field (NeRF) multilayer perceptron.

6 . The computer-implemented method of claim 5 , wherein the updating comprises modifying weights of the NeRF multilayer perceptron.

7 . The computer-implemented method of claim 2 , wherein the Langevin diffusion operation receives a conditioning input for generating a realistic 3D model as the updated original representation.

8 . The computer-implemented method of claim 7 , wherein the conditioning input comprises a text prompt describing a desired appearance of a rendered image of the 3D model.

9 . The computer-implemented method of claim 1 , wherein a level of the neutral noise is preserved during the applying.

10 . The computer-implemented method of claim 1 , wherein the representation comprises at least one of an image, a video, a panorama image, or audio data.

11 . The computer-implemented method of claim 1 , wherein at least one of the steps of incorporating, applying, extracting, and updating is performed on a server or in a data center to generate data, and the data is streamed to a user device.

12 . The computer-implemented method of claim 1 , wherein at least one of the steps of incorporating, applying, extracting, and updating is performed within a cloud computing environment.

13 . The computer-implemented method of claim 1 , wherein at least one of the steps of incorporating, applying, extracting, and updating is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

14 . The computer-implemented method of claim 1 , wherein at least one of the steps of incorporating, applying, extracting, and updating is performed on a virtual machine comprising a portion of a graphics processing unit.

15 . The computer-implemented method of claim 1 , wherein at least one of the steps of incorporating, applying, extracting, and updating is implemented to include advanced error correction, fault-tolerance, and self-healing capabilities.

16 . A synthesis system, comprising:
a memory that stores a representation; and a processor that is connected to the memory, wherein the processor is configured to update the representation by:
incorporating neutral noise into a rendering of the original representation to produce a latent representation;
applying a Langevin diffusion operation to the latent representation to modify the neutral noise based on a desired data distribution, producing a modified latent representation;
extracting a difference between a denoised latent produced by denoising the latent representation and a denoised modified latent produced by denoising the modified latent representation; and
updating the representation based on the difference.

17 . The system of claim 16 , wherein the representation is obtained by rendering an original representation, and further comprising updating the original representation based on the difference.

18 . The system of claim 17 , wherein the original representation is associated with a first domain and the representation x 0 is associated with a second domain.

19 . The system of claim 17 , wherein the original representation is defined by a neural radiance field (NeRF) multilayer perceptron.

20 . A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
incorporating neutral noise into a representation to produce a latent representation; applying a Langevin diffusion operation to the latent representation to modify the neutral noise based on a desired data distribution, producing a modified latent representation; extracting a difference between a denoised latent produced by denoising the latent representation and a denoised modified latent produced by denoising the modified latent representation; and updating the representation based on the difference.

21 . The non-transitory computer-readable media of claim 20 , wherein a level of the neutral noise is preserved during the applying.

Assignees

• Nvidia Corp

Inventors

• Chan Eric Ryan Wong
• Karras Tero Tapani
• Aittala Miika Samuli
• Aila Timo Oskari
• Laine Samuli Matias
• Chan Matthew Aaron Wong
• De Mello Shalini
• Nagano Koki

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title