Systems and methods for text-to-image generation using language models

Patent metadata

Field	Value
Publication number	US-12585919-B2
Application number	US-202318162535-A
Country	US
Kind code	B2
Filing date	Jan 31, 2023
Priority date	Oct 24, 2022
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

Embodiments described herein provide a mechanism for replacing existing text encoders in text-to-image generation models with more powerful pre-trained language models. Specifically, a translation network is trained to map features from the pre-trained language model output into the space of the target text encoder. The training preserves the rich structure of the pre-trained language model while allowing it to operate within the text-to-image generation model. The resulting modularized text-to-image model receives prompt and generates an image representing the features contained in the prompt.

First claim

What is claimed is:

1 . A method of integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model, the method comprising:
receiving, via a data interface, a text prompt describing image content; encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space; encoding, via the target text encoder, the prompt into a target embedding in a target representation space; transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space; generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding; computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding; computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding; training the translation network encoder and the translation network decoder based on the first loss and the second loss; and building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.

2 . The method of claim 1 , wherein the training comprises updating the parameters of the translation network encoder and the translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.

3 . The method of claim 1 , further comprising:
generating, by the image generator model, an image from the transformed source embedding.

4 . The method of claim 1 , further comprising:
receiving, via a data interface, an image associated with the text prompt; generating, via a conditioning network, a condition source embedding from the transformed source embedding; generating, by an image encoder, an image feature embedding in a latent image space from the image; generating a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding; generating a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding; generating, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding; computing a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and updating parameters of the denoising network and conditioning network based on the third loss.

5 . The method of claim 4 , wherein the updating comprises updating the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.

6 . The method of claim 1 , further comprising:
generating, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively; computing a third loss based, at least in part, on the target distribution and the transformed source distribution; and updating parameters of the discriminator network, the translation network encoder, and the translation network decoder based on the first loss, second loss, and third loss via backpropagation.

7 . The method of claim 6 , wherein the updating comprises updating the parameters of the discriminator network, the translation network encoder, and translation network decoder while keeping the pre-trained language model frozen.

8 . A system for integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model, the system comprising:
a communication interface that receives a plurality of training samples; a memory containing machine readable medium storing machine executable code; one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
receiving, via a data interface, a text prompt describing image content;
encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space;
encoding, via the target text encoder, the prompt into a target embedding in a target representation space;
transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding;
training the translation network encoder and the translation network decoder based on the first loss and the second loss; and
building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.

9 . The system of claim 8 , wherein the processor is further configured to train the translation network encoder and translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.

10 . The system of claim 8 , wherein the processor is further configured to:
generate, by the image generator model, an image from the transformed source embedding.

11 . The system of claim 8 , wherein the processor is further configured to:
receive, via a data interface, an image associated with the prompt; generate, via a conditioning network, a condition source embedding from the transformed source embedding; generate, by an image encoder, an image feature embedding in a latent image space from the image; generate a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding; generate a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding; generate, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding; compute a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and update parameters of the denoising network and conditioning network based on the third loss via backpropagation.

12 . The system of claim 11 , wherein to update parameters, the processor is further configured to update the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.

13 . The system of claim 8 , wherein the processor is further configured to:
generate, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively; compute a third loss based, at least in part, on the target distribution and the transfor

Assignees

• Salesforce Inc

Inventors

• Yu Ning
• Qin Can
• Xing Chen
• Zhang Shu
• Ermon Stefano
• Xiong Caiming
• Xu Ran

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title