Systems and methods for text-to-image generation using language models

Patent metadata

Field	Value
Publication number	US-2024185035-A1
Application number	US-202318162535-A
Country	US
Kind code	A1
Filing date	Jan 31, 2023
Priority date	Oct 24, 2022
Publication date	Jun 6, 2024
Grant date	—

Abstract

Embodiments described herein provide a mechanism for replacing existing text encoders in text-to-image generation models with more powerful pre-trained language models. Specifically, a translation network is trained to map features from the pre-trained language model output into the space of the target text encoder. The training preserves the rich structure of the pre-trained language model while allowing it to operate within the text-to-image generation model. The resulting modularized text-to-image model receives prompt and generates an image representing the features contained in the prompt.

First claim

What is claimed is:

1 . A method of integrating a pre-trained language model into a text-to-image model, the method comprising:
receiving, via a data interface, a prompt; encoding, via a pre-trained language model, the prompt into a source embedding in a source representation space; encoding, via a target text encoder, the prompt into a target embedding in a target representation space; transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space; generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding; computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding; computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding; and updating parameters of the translation network encoder and the translation network decoder based on the first loss and the second loss via backpropagation.

2 . The method of claim 1 , wherein the updating comprises updating the parameters of the translation network encoder and the translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.

3 . The method of claim 1 , further comprising:
generating, by an image generator, an image from the transformed source embedding.

4 . The method of claim 1 , further comprising:
receiving, via a data interface, an image associated with the prompt; generating, via a conditioning network, a condition source embedding from the transformed source embedding; generating, by an image encoder, an image feature embedding in a latent image space from the image; generating a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding; generating a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding; generating, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding; computing a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and updating parameters of the denoising network and conditioning network based on the third loss via backpropagation.

5 . The method of claim 4 , wherein the updating comprises updating the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.

6 . The method of claim 1 , further comprising:
generating, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively; computing a third loss based, at least in part, on the target distribution and the transformed source distribution; and updating parameters of the discriminator network, the translation network encoder, and the translation network decoder based on the first loss, second loss, and third loss via backpropagation.

7 . The method of claim 6 , wherein the updating comprises updating the parameters of the discriminator network, the translation network encoder, and translation network decoder while keeping the pre-trained language model frozen.

8 . A system for integrating a pre-trained language model into a text-to-image model, the system comprising:
a communication interface that receives a plurality of training samples; a memory containing machine readable medium storing machine executable code; one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
receive, via a data interface, a prompt;
encode, via a pre-trained language model, the prompt into a source embedding in a source representation space;
encode, via a target text encoder, the prompt into a target embedding in a target representation space;
transform, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
generate, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
compute a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
compute a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding; and
update parameters of the translation network encoder and the translation network decoder based on the first loss and the second loss via backpropagation.

9 . The system of claim 8 , wherein to update parameters, the processor is further configured to update parameters of the translation network encoder and translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.

10 . The system of claim 8 , wherein the processor is further configured to:
generate, by an image generator, an image from the transformed source embedding.

11 . The system of claim 8 , wherein the processor is further configured to:
receive, via a data interface, an image associated with the prompt; generate, via a conditioning network, a condition source embedding from the transformed source embedding; generate, by an image encoder, an image feature embedding in a latent image space from the image; generate a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding; generate a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding; generate, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding; compute a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and update parameters of the denoising network and conditioning network based on the third loss via backpropagation.

12 . The system of claim 11 , wherein to update parameters, the processor is further configured to update the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.

13 . The system of claim 8 , wherein the processor is further configured to:
generate, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively; compute a third loss based, at least in part, on the target distribution and the transformed source distribution; and update parameters of the discriminator network, the translation network encoder, and the translation network decoder based on the first loss, second loss, and third loss via backpropagation.

14 . The system of claim 13 , wherein to update parameters, the processor is further configured to update parameters of the discriminator network, the translation network encoder, and the translation network decoder while keeping the pre-trai

Assignees

• Salesforce Inc

Inventors

• Yu Ning
• Qin Can
• Xing Chen
• Zhang Shu
• Ermon Stefano
• Xiong Caiming
• Xu Ran

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06T5/002

  Physics · mapped topic