Utilizing a diffusion neural network for mask aware image and typography editing

Patent metadata

Field	Value
Publication number	US-12536722-B2
Application number	US-202318303898-A
Country	US
Kind code	B2
Filing date	Apr 20, 2023
Priority date	Apr 20, 2023
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a diffusion neural network for mask aware image and typography editing. For example, in one or more embodiments the disclosed systems utilize a text-image encoder to generate a base image embedding from a base digital image. Moreover, the disclosed systems generate a mask-segmented image by combining a shape mask with the base digital image. In one or more implementations, the disclosed systems utilize noising steps of a diffusion noising model to generate a mask-segmented image noise map from the mask-segmented image. Furthermore, the disclosed systems utilize a diffusion neural network to create a stylized image corresponding to the shape mask from the base image embedding and the mask-segmented image noise map.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
generating, from a base digital image utilizing a trained text-image encoder, a base image embedding comprising a vector representation of the base digital image; generating a mask-segmented image by segmenting the base digital image utilizing a shape mask; generating, utilizing noising steps of a diffusion noising model, a mask-segmented image noise map from the mask-segmented image; and generating, utilizing a diffusion neural network, a stylized image corresponding to the shape mask in a style of the base digital image by denoising the mask-segmented image noise map utilizing denoising layers of the diffusion neural network conditioned on the base image embedding.

2 . The computer-implemented method of claim 1 , further comprising generating the shape mask from a typography character and wherein creating the stylized image comprises generating a stylized typography character that reflects the base digital image utilizing the diffusion neural network.

3 . The computer-implemented method of claim 1 , wherein generating, utilizing the noising steps of the diffusion noising model, the mask-segmented image noise map comprises utilizing reverse diffusion steps of a reverse diffusion neural network to generate the mask-segmented image noise map from the mask-segmented image.

4 . The computer-implemented method of claim 1 , wherein creating the stylized image comprises generating an intermediate noise map from the base image embedding utilizing a denoising step of the diffusion neural network conditioned on the base image embedding.

5 . The computer-implemented method of claim 4 , wherein creating the stylized image comprises generating the stylized image from the intermediate noise map utilizing additional denoising steps of the diffusion neural network conditioned on the base image embedding.

6 . The computer-implemented method of claim 1 , further comprising generating a stylized animation by generating a plurality of stylized images utilizing a first structural number of steps for a first frame of the stylized animation and a second structural number of steps for a second frame of the stylized animation.

7 . The computer-implemented method of claim 1 , wherein generating the mask-segmented image noise map comprises:
selecting a structural number of steps based on user interaction with a client device; and utilizing the structural number of steps of the diffusion noising model to generate the mask-segmented image noise map from the mask-segmented image.

8 . The computer-implemented method of claim 7 , wherein creating the stylized image comprises utilizing the structural number of steps of the diffusion neural network to create the stylized image from the base image embedding and the mask-segmented image noise map.

9 . The computer-implemented method of claim 1 , further comprising generating the base image embedding by:
receiving, from a client device, an edit text; and generating the base digital image from the edit text.

10 . A system comprising:
one or more memory devices comprising a base digital image, a typography character mask, a trained text-image encoder, and a diffusion neural network; and one or more processors configured to cause the system to:
generate, from the base digital image, a base image embedding comprising a vector representation of the base digital image;
generate, utilizing noising steps of a diffusion noising model, a mask-segmented image noise map from a mask-segmented image generated by segmenting the base digital image utilizing the typography character mask; and
generate, utilizing the diffusion neural network, a stylized typography character in a style of the base digital image from the mask-segmented image noise map by conditioning denoising steps of the diffusion neural network utilizing the base image embedding.

11 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to generate the base image embedding from the base digital image utilizing the trained text-image encoder.

12 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to generate an additional stylized typography character utilizing an additional structural number of noising steps.

13 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to:
generate a mask-segmented image from the typography character mask and the base digital image; and generate the mask-segmented image noise map by utilizing reverse diffusion steps of a reverse diffusion neural network to generate the mask-segmented image noise map from the mask-segmented image.

14 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to:
select a structural number of steps; and create the stylized typography character from the base image embedding by utilizing the structural number of steps of the diffusion neural network.

15 . A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating, from a base digital image, a base image embedding comprising a vector representation of the base digital image; determining a structural number of steps based on user interaction with a user interface of a client device; generating, utilizing the structural number of steps of a diffusion noising model, a mask-segmented image noise map from a mask-segmented image generated by segmenting the base digital image utilizing a shape mask; and generating, utilizing the structural number of steps of a diffusion neural network, a stylized image corresponding to the shape mask in a style of the base digital image by denoising the mask-segmented image noise map utilizing denoising layers of the diffusion neural network conditioned on the base image embedding.

16 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
generating a mask-segmented digital image by applying the shape mask to the base digital image; and generating the mask-segmented image noise map from the mask-segmented digital image utilizing the structural number of steps of the diffusion noising model.

17 . The non-transitory computer readable medium of claim 15 , wherein the shape mask comprises a typography character mask and the operations further comprise creating the stylized image by generating a stylized typography character that reflects the base digital image utilizing the structural number of steps of the diffusion neural network.

18 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise generating a stylized animation by generating a plurality of style images utilizing a plurality of structural numbers of steps.

19 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
generating an intermediate noise map from the mask-segmented image noise map utilizing a denoising step of the diffusion neural network conditioned on the base image embedding; and generating the stylized image from the intermediate noise map utilizing additional denoising steps of the diffusion neural network conditioned on the base image embedding.

20 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
provi

Assignees

• Adobe Inc

Inventors

• AGGARWAL Pranav
• Ravi Hareesh
• Harikumar Midhun
• Kale Ajinkya Gorakhnath
• Chen Fengbin
• Marri Venkata Naveen Kumar Yadav

Classifications

• G06T2207/20212

  Image combination · CPC title

• G06T2207/20092

  Interactive image processing based on input by user · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title