Text-Based Real Image Editing with Diffusion Models

Patent metadata

Field	Value
Publication number	US-2024355017-A1
Application number	US-202318302508-A
Country	US
Kind code	A1
Filing date	Apr 18, 2023
Priority date	Apr 18, 2023
Publication date	Oct 24, 2024
Grant date	—

Abstract

Methods and systems for editing an image are disclosed herein. The method includes receiving an input image and a target text, the target text indicating a desired edit for the input image and obtaining, by the computing system, a target text embedding based on the target text. The method also includes obtaining, by the computing system, an optimized text embedding based on the target text embedding and the input image and fine-tuning, by the computing system, a diffusion model based on the optimized text embedding. The method can further include interpolating, by the computing system, the target text embedding and the optimized text embedding to obtain an interpolated embedding and generating, by the computing system, an edited image including the desired edit using the diffusion model based on the input image and the interpolated embedding.

First claim

What is claimed is:

1 . A computer-implemented method for editing an image, the method comprising:
receiving, by a computing system, an input image and a target text, the target text indicating a desired edit for the input image; obtaining, by the computing system, a target text embedding based on the target text; obtaining, by the computing system, an optimized text embedding based on the target text embedding and the input image; fine-tuning, by the computing system, a diffusion model based on the optimized text embedding; interpolating, by the computing system, the target text embedding and the optimized text embedding to obtain an interpolated embedding; and generating, by the computing system, an edited image including the desired edit using the diffusion model based on the input image and the interpolated embedding.

2 . The computer-implemented method of claim 1 , wherein obtaining the target text embedding includes
providing, by the computing system, the target text to a text encoder; and receiving, by the computing system, the target text embedding from the text encoder.

3 . The computer-implemented method of claim 1 , wherein obtaining the optimized text embedding includes
freezing, by the computing system, the parameters of the diffusion model; optimizing, by the computing system, the target text embedding using a denoising diffusion objective to obtain the optimized text embedding; and outputting the optimized text embedding, wherein the optimized text embedding is a text embedding that matches the input image.

4 . The computer-implemented method of claim 1 , wherein fine-tuning the diffusion model includes
freezing, by the computing system, the optimized text embedding; and optimizing, by the computing system, at least one model parameter of the diffusion model using a loss function.

5 . The computer-implemented method of claim 4 , wherein optimizing the at least one model parameter includes conditioning the diffusion model on the optimized text embedding.

6 . The computer-implemented method of claim 4 , wherein fine-tuning the diffusion model further includes
fine-tuning, by the computing system, at least one auxiliary diffusion model, wherein fine-tuning the at least one auxiliary diffusion model includes
freezing, by the computing system, the target text embedding; and
optimizing, by the computing system, the at least one auxiliary diffusion model using the loss function.

7 . The computer-implemented method of claim 6 , wherein optimizing the at least one auxiliary diffusion model includes conditioning the at least one auxiliary diffusion model on the target text embedding.

8 . The computer-implemented method of claim 6 , wherein generating the edited image includes
generating, by the computing system, a low-resolution version of the edited image using the diffusion model; super-resolving, by the computing system, the low-resolution version of the edited image into a final high-resolution version of the edited image using the at least one auxiliary diffusion model.

9 . The computer-implemented method of claim 1 , wherein generating the edited image includes
generating, by the computing system, the edited image using the diffusion model conditioned on the interpolated embedding.

10 . A computing system for editing images, the computing system comprising:
one or more processors; and a non-transitory, computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
receiving an input image and a target text, the target text indicating a desired edit for the input image;
obtaining a target text embedding based on the target text;
obtaining an optimized text embedding based on the target text embedding and the input image fine-tuning a diffusion model based on the optimized text embedding;
interpolating the target text embedding and the optimized text embedding to obtain an interpolated embedding; and
generating an edited image including the desired edit using the diffusion model based on the input image and the interpolated embedding.

11 . The computing system of claim 10 , wherein obtaining the target text embedding includes providing the target text to a text encoder; and
receiving the target text embedding from the text encoder.

12 . The computing system of claim 10 , wherein obtaining the optimized text embedding includes
freezing, by the computing system, the parameters of the diffusion model; optimizing, by the computing system, the target text embedding using a denoising diffusion objective to obtain the optimized text embedding; and outputting the optimized text embedding, wherein the optimized text embedding is a text embedding that matches the input image.

13 . The computing system of claim 10 , wherein fine-tuning the diffusion model includes
freezing the optimized text embedding; and optimizing at least one model parameter of the diffusion model using a loss function.

14 . The computing system of claim 13 , wherein optimizing the at least one model parameter includes conditioning the diffusion model on the optimized text embedding.

15 . The computing system of claim 13 , wherein fine-tuning the diffusion model further includes
fine-tuning at least one auxiliary diffusion model, wherein fine-tuning the at least one auxiliary diffusion model includes
freezing the target text embedding; and
optimizing the at least one auxiliary diffusion model using the loss function.

16 . The computing system of claim 15 , wherein optimizing the at least one auxiliary diffusion model includes conditioning the at least one auxiliary diffusion model on the target text embedding.

17 . The computer-implemented method of claim 15 , wherein generating the edited image includes
generating a low-resolution version of the edited image using the diffusion model; super-resolving the low-resolution version of the edited image into a final high-resolution version of the edited image using the at least one auxiliary diffusion model.

18 . The computing system of claim 10 , wherein generating the edited image includes
generating the edited image using the diffusion model conditioned on the interpolated embedding.

19 . A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
receiving an input image and a target text, the target text indicating a desired edit for the input image; obtaining a target text embedding based on the target text; obtaining an optimized text embedding based on the target text embedding and the input image; fine-tuning a diffusion model based on the optimized text embedding; interpolating the target text embedding and the optimized text embedding to obtain an interpolated embedding; and generating an edited image including the desired edit using the diffusion model based on the input image and the interpolated embedding.

20 . The non-transitory, computer-readable medium of claim 19 , wherein generating the edited image includes
generating the edited image using the diffusion model conditioned on the interpolated embedding.

Assignees

• Google Llc

Inventors

• Elyahu Zada Shiran
• Kawar Bahjat
• Lang Oran
• Tov Omer
• Chang Huiwen
• Dekel Tali
• Mosseri Inbar

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06F40/279

  Recognition of textual entities · CPC title