Text-to-image diffusion model rearchitecture

Patent metadata

Field	Value
Publication number	US-12469273-B2
Application number	US-202318400873-A
Country	US
Kind code	B2
Filing date	Dec 29, 2023
Priority date	May 26, 2023
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Described is a system for improving machine learning models. In some cases, the system improves such models by identifying a performance characteristic for machine learning model blocks in an iterative denoising process of a machine learning model, connecting a prior machine learning model block with a subsequent machine learning model block of the machine learning model blocks within the machine learning model based on the identified performance characteristic, identifying a prompt of a user, the prompt indicative of an intent of the user for generative images, and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

First claim

What is claimed is:

1 . A system comprising:
at least one processor; and at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
identifying a performance characteristic for individual machine learning model blocks in an iterative denoising process of a machine learning model, wherein the performance characteristic includes a latency characteristic indicative of a time delay for the corresponding machine learning model block to process an input to generate an output, wherein the operations further comprise identifying the current machine learning model block based on the latency characteristic, wherein the current machine learning model block is subsequent to the prior machine learning model block and the current machine learning model block is prior to the subsequent machine learning model block;
identifying a current machine learning model block to remove from the machine learning model based on the identified performance characteristic;
identifying a prior machine learning model block connected to an input of the current machine learning model block and a subsequent machine learning model block connected to an output of the current machine learning model block;
identifying a prompt of a user, the prompt indicative of an intent of the user for generative images; and
analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

2 . The system of claim 1 , wherein the machine learning model includes a stable diffusion model, wherein the machine learning model blocks include cross-attention blocks.

3 . The system of claim 1 , wherein the operations further comprise adding a new machine learning model block to the machine learning model.

4 . The system of claim 3 , wherein the new machine learning model block is a copy of an existing machine learning model block of the machine learning model.

5 . The system of claim 3 , wherein the operations further comprise disconnecting a connection between a first and second block, and generating a connection between the first block with the new block and the new block with the second block.

6 . The system of claim 3 , wherein the machine learning model includes a stable diffusion model, wherein the machine learning model blocks include ResNet blocks.

7 . The system of claim 3 , wherein the new machine learning model block is added subsequent to the removal of the current machine learning model block.

8 . The system of claim 3 , wherein the operations further comprise adding the new machine learning model block at different locations in the machine learning model and assessing the performance of the machine learning model with the new machine learning model block at each of the different locations, and adding the new machine learning model block at a particular location based on the assessed performance.

9 . The system of claim 1 , wherein the operations further comprise continuously removing blocks and continuously adding new blocks until a first desired performance characteristic threshold is met when removing blocks and a second desired performance characteristic threshold is met when adding blocks.

10 . The system of claim 1 , wherein the machine learning model includes a stable diffusion model, wherein the machine learning model blocks include ResNet blocks.

11 . The system of claim 1 , wherein the operations further comprise disconnecting the prior machine learning model block with the current machine learning model block and disconnecting the current machine learning model block with the subsequent machine learning model block.

12 . The system of claim 11 , wherein the operations further comprise deleting the current machine learning model block after the current machine learning model block is disconnected from the prior machine learning model block and the subsequent machine learning model block.

13 . The system of claim 1 , wherein connecting the prior machine learning model block with the subsequent machine learning model block of the machine learning model blocks is in response to identifying the performance characteristic for individual machine learning model blocks, wherein in response to identifying the performance characteristic for individual machine learning model blocks, the operations further comprise connecting another prior machine learning model block with another subsequent machine learning model block for another node.

14 . The system of claim 1 , wherein the performance characteristic further includes a computational complexity characteristic indicative of a computational requirement for the corresponding machine learning model block.

15 . The system of claim 1 , wherein the performance characteristic further includes a memory characteristic indicative of a memory storage requirement for the corresponding machine learning model block.

16 . The system of claim 1 , wherein the iterative denoising process includes adding random noise and an output image generated during a previous iteration to the machine learning model causing the generation of the output image of the current iteration.

17 . The system of claim 16 , wherein the output of the machine learning model is processed through a decoder to generate the output image of the current iteration.

18 . The system of claim 1 , wherein the operations further comprise: modifying the machine learning model to generate a modified machine learning model where the current machine learning model block is skipped by generating a new connection between connecting one or more output nodes of the prior machine learning model block with one or more input nodes of the subsequent machine learning model block, wherein analyzing the data using the machine learning model comprises analyzing the data using the modified machine learning model.

19 . A method comprising:
identifying a performance characteristic for individual machine learning model blocks in an iterative denoising process of a machine learning model, wherein the performance characteristic includes a latency characteristic indicative of a time delay for the corresponding machine learning model block to process an input to generate an output, wherein the operations further comprise identifying the current machine learning model block based on the latency characteristic, wherein the current machine learning model block is subsequent to the prior machine learning model block and the current machine learning model block is prior to the subsequent machine learning model block; identifying a current machine learning model block to remove from the machine learning model based on the identified performance characteristic; identifying a prior machine learning model block connected to an input of the current machine learning model block and a subsequent machine learning model block connected to an output of the current machine learning model block; identifying a prompt of a user, the prompt indicative of an intent of the user for generative images; and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

20 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a

Assignees

• Snap Inc

Inventors

• Chemerys Pavlo
• Eles Colin
• Hu Ju
• Jin Qing
• Li Yanyu
• Muca Ergeta
• Ren Jian
• Sagar Dhritiman
• Stoliar Aleksei
• Tulyakov Sergey
• Wang Huan

Classifications

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title