Implementing portrait editing using a machine learning model

Patent metadata

Field	Value
Publication number	US-2026011056-A1
Application number	US-202418763728-A
Country	US
Kind code	A1
Filing date	Jul 3, 2024
Priority date	Jul 3, 2024
Publication date	Jan 8, 2026
Grant date	—

Abstract

The present disclosure describes techniques for implementing portrait editing using a machine learning model. An and a text prompt are input into a first machine learning model. The image comprises a portrait of a subject. The text prompt indicates a target result of editing the image. The first machine learning model is trained to perform portrait editing while preserving untargeted features. An editing mask is generated by the first machine-learning model based on the image. The editing mask indicates a first area for editing and a second area for preserving original content of the image. A mask-guided predicted noise is computed at each timestep and a process of editing the image is guided by the first machine learning model based on the editing mask. An edited image is generated by the first machine learning model. The edited image comprises the target editing result and retains detailed features of the subject.

First claim

What is claimed is:

1 . A method of implementing portrait editing using a machine learning model, comprising:
inputting an image and a text prompt into a first machine learning model, wherein the image comprises a portrait of a subject, wherein the text prompt indicates a target result of editing the image, and wherein the first machine learning model is trained to perform portrait editing while preserving untargeted features; generating an editing mask by the first machine-learning model based on the image, wherein the editing mask indicates a first area for editing and a second area for preserving original content of the image; computing a mask-guided predicted noise at each timestep and guiding a process of editing the image by the first machine learning model based on the editing mask; and generating an edited image by the first machine learning model, wherein the edited image comprises the target editing result and retains detailed features of the subject.

2 . The method of claim 1 , further comprising:
generating training pairs by a second machine learning model, wherein the training pairs are utilized to train the first machine learning model, wherein the training pairs align with a specified editing direction, wherein each training pair comprises a source image and a target image, and wherein the source image and the target image in each training pair comprise a same subject and indicate the specified editing direction.

3 . The method of claim 2 , further comprising:
generating each training pair through a single denoising process by the second machine learning model to enhance identity consistency in the source image and the target image; and generating a single image by the single denoising process, wherein the single image comprises a horizontal concatenation of the source image and the target image.

4 . The method of claim 3 , further comprising:
guiding the single denoising process using a pose image to ensure spatial alignment by featuring a same pose in a left and right parts of the single image.

5 . The method of claim 3 , further comprising:
generating identity embeddings based on a real-world portrait image; and guiding the single denoising process using the identity embeddings.

6 . The method of claim 5 , further comprising:
providing the identity embeddings to the single denoising process by combining the identity embeddings with text embeddings computed from prompts depicting the single image.

7 . The method of claim 2 , further comprising:
generating the training pairs to cover a diverse range of appearances by utilizing diverse real-world portrait images.

8 . The method of claim 2 , further comprising:
training the first machine learning model using the training pairs, wherein the first machine learning model learns pertinent information from the training pairs, and wherein the pertinent information indicates the specified editing direction and preservation of untargeted subject features.

9 . The method of claim 8 , further comprising:
generating spatial embeddings based on the source image in each training pair; concatenating the spatial embeddings with a noisy latent to generate a first concatenation; and inputting the first concatenation into the first machine learning model.

10 . The method of claim 9 , further comprising:
generating target text embeddings based on a target prompt depicting the target image in each training pair; generating image embeddings based on the source image in each training pair and projecting the image embeddings to a space of text embeddings, wherein the image embeddings indicate visual information derived from the source image; concatenating the target text embeddings and the image embeddings to generate a second concatenation; and inputting the second concatenation into a cross-attention layer of the first machine learning model.

11 . The method of claim 10 , further comprising:
enabling the first machine learning model to possess reconstruction capabilities of reconstructing input images by replacing the target text embeddings with source text embeddings and replacing the target image with the source image in a predetermined percentage of time during training, wherein the source text embeddings are generated based on a source prompt depicting the source image in each training pair, and wherein the reconstruction capabilities of the first machine learning model is utilized during an inference phase for mask generation.

12 . A system of implementing portrait editing using a machine learning model, comprising:
at least one processor; and at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations comprising: inputting an image and a text prompt into a first machine learning model, wherein the image comprises a portrait of a subject, wherein the text prompt indicates a target result of editing the image, and wherein the first machine learning model is trained to perform portrait editing while preserving untargeted features; generating an editing mask by the first machine-learning model based on the image, wherein the editing mask indicates a first area for editing and a second area for preserving original content of the image; computing a mask-guided predicted noise at each timestep and guiding a process of editing the image by the first machine learning model based on the editing mask; and generating an edited image by the first machine learning model, wherein the edited image comprises the target editing result and retains detailed features of the subject.

13 . The system of claim 12 , the operations further comprising:
generating training pairs by a second machine learning model, wherein the training pairs are utilized to train the first machine learning model, wherein the training pairs align with a specified editing direction, wherein each training pair comprises a source image and a target image, and wherein the source image and the target image in each training pair comprise a same subject and indicate the specified editing direction.

14 . The system of claim 13 , the operations further comprising:
generating each training pair through a single denoising process by the second machine learning model to enhance identity consistency in the source image and the target image; and generating a single image by the single denoising process, wherein the single image comprises a horizontal concatenation of the source image and the target image.

15 . The system of claim 13 , the operations further comprising:
training the first machine learning model using the training pairs, wherein the first machine learning model learns pertinent information from the training pairs, and wherein the pertinent information indicates the specified editing direction and preservation of untargeted subject features.

16 . The system of claim 15 , the operations further comprising:
generating spatial embeddings based on the source image in each training pair; concatenating the spatial embeddings with a noisy latent to generate a first concatenation; generating target text embeddings based on a target prompt depicting the target image in each training pair; generating image embeddings based on the source image in each training pair and projecting the image embeddings to a space of text embeddings; concatenating the target text embeddings and the image embeddings to generate a second concatenation; and inputting the first

Assignees

• Lemon Inc

Inventors

• Zhi Tiancheng
• Chen Bowei
• Zhu Peihao
• Sang Shen
• Liu Jing
• Luo Linjie

Classifications

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title