Trajectory stitching for accelerating diffusion models

Patent metadata

Field	Value
Publication number	US-2025103968-A1
Application number	US-202418821611-A
Country	US
Kind code	A1
Filing date	Aug 30, 2024
Priority date	Sep 26, 2023
Publication date	Mar 27, 2025
Grant date	—

Abstract

Diffusion models are machine learning algorithms that are uniquely trained to generate high-quality data from an input lower-quality data. Diffusion probabilistic models use discrete-time random processes or continuous-time stochastic differential equations (SDEs) that learn to gradually remove the noise added to the data points. With diffusion probabilistic models, high quality output currently requires sampling from a large diffusion probabilistic model which corners at a high computational cost. The present disclosure stitches together the trajectory of two or more inferior diffusion probabilistic models during a denoising process, which can in turn accelerate the denoising process by avoiding use of only a single large diffusion probabilistic model.

First claim

What is claimed is:

1 . A method, comprising:
at a device: denoising an input over a first sequence of steps using a first diffusion probabilistic model to generate a first denoised version of the input; denoising the first denoised version of the input over a second sequence of steps using a second diffusion probabilistic model to generate a second denoised version of the input, wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model in at least one respect; and outputting the second denoised version of the input.

2 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are pretrained diffusion probabilistic models in a same model family.

3 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained on a same training dataset.

4 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained on a same data distribution.

5 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model have a same architecture.

6 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model have different architectures.

7 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained to perform a same task.

8 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are configured with a same input and output shape.

9 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are stylized stable diffusion models.

10 . The method of claim 1 , wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model as a result of the first diffusion probabilistic model being of a smaller size than the second diffusion probabilistic model.

11 . The method of claim 1 , wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model as a result of the first diffusion probabilistic model having fewer parameters than the second diffusion probabilistic model.

12 . The method of claim 1 , wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model as a result of the first diffusion probabilistic model being configured with less complexity than the second diffusion probabilistic model.

13 . The method of claim 12 , wherein the first diffusion probabilistic model includes fewer floating-point operations (FLOPs) than the second diffusion probabilistic model.

14 . The method of claim 1 , wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model as a result of the second diffusion probabilistic model being a finetuned version of the first diffusion probabilistic model.

15 . The method of claim 1 , wherein the first diffusion probabilistic model that is inferior to the second diffusion probabilistic model in at least one respect is further compressed.

16 . The method of claim 15 , wherein the first diffusion probabilistic model is compressed by reducing a number of steps it takes.

17 . The method of claim 15 , wherein the first diffusion probabilistic model is compressed by reducing its computational cost.

18 . The method of claim 1 , wherein the first sequence of steps and the second sequence of steps are steps of a denoising process.

19 . The method of claim 1 , wherein the first sequence of steps generates lower frequency components from the input than the second sequence of steps.

20 . The method of claim 1 , wherein a last step in the first sequence of steps outputs to a first step in the second sequence of steps.

21 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are preselected for processing the input.

22 . The method of claim 1 , wherein a number of steps in the first sequence of steps and a number of steps in the second sequence of steps are preconfigured.

23 . The method of claim 1 , wherein the input is a noisy image.

24 . The method of claim 1 , wherein the input is a noisy audio.

25 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained for text-to-image generation.

26 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained for audio synthesis.

27 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained for three-dimensional (3D) content generation.

28 . The method of claim 1 , wherein the first diffusion probabilistic model and the second diffusion probabilistic model are trained for text-to-video generation.

29 . The method of claim 1 , wherein the second denoised version of the input is output to a third diffusion probabilistic model.

30 . The method of claim 29 , wherein the third diffusion probabilistic model denoises the second denoised version of the input over a third sequence of steps to generate a third denoised version of the input, and wherein the third denoised version of the input is output.

31 . The method of claim 29 , wherein the second diffusion probabilistic model is inferior to the third diffusion probabilistic model in at least one respect.

32 . A system, comprising:
a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions to: denoise an input over a first sequence of steps using a first diffusion probabilistic model to generate a first denoised version of the input; denoise the first denoised version of the input over a second sequence of steps using a second diffusion probabilistic model to generate a second denoised version of the input, wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model in at least one respect; and output the second denoised version of the input.

33 . The system of claim 32 , wherein the first diffusion probabilistic model is inferior to the second diffusion probabilistic model as a result of at least one of:
the first diffusion probabilistic model being of a smaller size than the second diffusion probabilistic model, the first diffusion probabilistic model having fewer parameters than the second diffusion probabilistic model, the first diffusion probabilistic model being configured with less complexity than the second diffusion probabilistic model, or the second diffusion probabilistic model being a finetuned version of the first diffusion probabilistic model.

34 . The system of claim 32 , wherein the input is one of:
a noisy image, or a noisy audio.

35 . The system of claim 32 , wherein the first diffusion probabilistic model and the second diffusio

Assignees

• Nvidia Corp

Inventors

• Pan Zizheng
• Huang De-An
• Nie Weili
• Yu Zhiding
• Xiao Chaowei
• Anandkumar Anima

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N20/20 Primary

  Ensemble learning · CPC title