Generating and modifying digital images using a joint feature style latent space of a generative neural network

Patent metadata

Field	Value
Publication number	US-12586270-B2
Application number	US-202217655739-A
Country	US
Kind code	B2
Filing date	Mar 21, 2022
Priority date	Mar 21, 2022
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for latent-based editing of digital images using a generative neural network. In particular, in one or more embodiments, the disclosed systems perform latent-based editing of a digital image by mapping a feature tensor and a set of style vectors for the digital image into a joint feature style space. In one or more implementations, the disclosed systems apply a joint feature style perturbation and/or modification vectors within the joint feature style space to determine modified style vectors and a modified feature tensor. Moreover, in one or more embodiments the disclosed systems generate a modified digital image utilizing a generative neural network from the modified style vectors and the modified feature tensor.

First claim

What is claimed is:

1 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to perform operations comprising:
generating style vectors representing a digital image; generating a feature tensor from a first subset of the style vectors utilizing a series of generative convolutional blocks of a generative neural network; mapping a second subset of the style vectors and the feature tensor to a joint feature style space by combining the second subset of the style vectors with the feature tensor to generate a joint feature style vector, the second subset of the style vectors including one or more style vectors not utilized in generating the feature tensor; determining a modified joint feature style vector by combining a joint feature style perturbation with the joint feature style vector within the joint feature style space; extracting modified style vectors and a modified feature tensor from the modified joint feature style vector; and generating a modified digital image, utilizing the generative neural network, from the modified style vectors and the modified feature tensor.

2 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising:
generating intermediate latent vectors from the digital image utilizing a plurality of convolutional layers to map an initial latent code corresponding to the digital image to an intermediate latent space; and generating the style vectors from the intermediate latent vectors utilizing a plurality of learned affine transformations to embed the intermediate latent vectors within a latent style space.

3 . The non-transitory computer-readable medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising:
receiving user input of a digital image modification corresponding to the digital image; determining an image modification vector corresponding to the digital image modification in at least one of the intermediate latent space corresponding to the intermediate latent vectors or the latent style space corresponding to the style vectors; and mapping, utilizing the series of generative convolutional blocks of the generative neural network, the image modification vector from the at least one of the intermediate latent space or the latent style space to the joint feature style space to generate a joint feature style modification vector.

4 . The non-transitory computer-readable medium of claim 3 , wherein determining the modified joint feature style vector further comprises combining the joint feature style perturbation and the joint feature style modification vector with the joint feature style vector within the joint feature style space.

5 . The non-transitory computer-readable medium of claim 4 , wherein determining the modified style vectors and the modified feature tensor further comprises:
combining the joint feature style perturbation and the joint feature style modification vector with the joint feature style vector within the joint feature style space to generate the modified joint feature style vector; and extracting the modified style vectors and the modified feature tensor from the modified joint feature style vector.

6 . The non-transitory computer-readable medium of claim 1 , wherein:
the series of generative convolutional blocks of the generative neural network comprises a first subset of generative convolutional blocks and a second subset of generative convolutional blocks, and generating the feature tensor comprises utilizing the first subset of generative convolutional blocks to generate the feature tensor from the first subset of the style vectors.

7 . The non-transitory computer-readable medium of claim 6 , wherein generating the modified digital image comprises utilizing the second subset of generative convolutional blocks of the generative neural network to generate the modified digital image from the modified style vectors and the modified feature tensor.

8 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising determining the joint feature style perturbation by utilizing a gradient descent optimization model to reduce a reconstruction error between the digital image and a reconstructed digital image.

9 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising determining the joint feature style perturbation utilizing a neural network encoder comprising learned parameters tuned to predict joint feature style perturbations within the joint feature style space from digital images.

10 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising determining the joint feature style perturbation utilizing a locality regularization term.

11 . A system comprising:
at least one storage device comprising: a digital image, an encoder comprising learned transformations, and a generative neural network comprising generative convolutional blocks; and at least one processor configured to cause the system to:
generate style vectors for a first digital image from intermediate latent vectors utilizing the learned transformations of the encoder;
determine a feature tensor from a first subset of the style vectors utilizing the generative convolutional blocks of the generative neural network;
determine a joint feature style vector within a joint feature style space by combining a second subset of the style vectors and the feature tensor, the second subset of the style vectors including one or more style vectors not utilized in generating the feature tensor;
generate a modified joint feature style vector by combining a joint feature style perturbation with the joint feature style vector within the joint feature style space; and
generate a second digital image, utilizing the generative neural network, from the modified joint feature style vector.

12 . The system of claim 11 , wherein the at least one processor is further configured to cause the system to determine the feature tensor from the first subset of the style vectors utilizing a first subset of the generative convolutional blocks.

13 . The system of claim 12 , wherein the at least one processor is further configured to cause the system to generate the second digital image from the modified joint feature style vector utilizing a second subset of the generative convolutional blocks, the second subset of the generative convolutional blocks including one or more generative convolutional blocks not utilized in generating the feature tensor from the first subset of the style vectors.

14 . The system of claim 11 , wherein the at least one processor is further configured to modify the second digital image by:
mapping one or more image modification vectors from an intermediate latent space corresponding to the intermediate latent vectors to a style space corresponding to the style vectors utilizing the learned transformations of the encoder; mapping, utilizing the generative convolutional blocks of the generative neural network, the one or more image modification vecto

Assignees

• Adobe Inc

Inventors

• Qu Hui
• Faieta Baldo
• Smith Cameron
• Shechtman Elya
• Lu Jingwan
• Kalarot Ratheesh
• Zhang Richard
• Motiian Saeid
• Ghadar Shabnam
• Lin Wei-An

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/045

  Combinations of networks · CPC title