Attribute conditioned image generation

Patent metadata

Field	Value
Publication number	US-11640684-B2
Application number	US-202016934858-A
Country	US
Kind code	B2
Filing date	Jul 21, 2020
Priority date	Jul 21, 2020
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

A method, apparatus, and non-transitory computer readable medium for image processing are described. Embodiments of the method, apparatus, and non-transitory computer readable medium include identifying an original image including a plurality of semantic attributes, wherein each of the semantic attributes represents a complex set of features of the original image; identifying a target attribute value that indicates a change to a target attribute of the semantic attributes; computing a modified feature vector based on the target attribute value, wherein the modified feature vector incorporates the change to the target attribute while holding at least one preserved attribute of the semantic attributes substantially unchanged; and generating a modified image based on the modified feature vector, wherein the modified image includes the change to the target attribute and retains the at least one preserved attribute from the original image.

First claim

What is claimed is:

1. A method for image processing, comprising:
identifying an original image including a plurality of semantic attributes, wherein a target attribute of the plurality of semantic attributes represents a complex set of features of the original image;
identifying a target attribute value that indicates a change to the target attribute of the plurality of semantic attributes;
computing a modified feature vector using a mapping network comprising a continuous normalizing flow (CNF) block that includes a non-linear activation function and that takes the target attribute value as input, wherein the modified feature vector incorporates the change to the target attribute while holding at least one preserved attribute of the plurality of semantic attributes unchanged; and
generating a modified image based on the modified feature vector using a generator network, wherein the modified image includes the change to the target attribute and retains the at least one preserved attribute from the original image.

2. The method of claim 1 , further comprising:
identifying an original feature vector representing the original image;
identifying a plurality of original attribute values corresponding to original attributes based on the original image; and
computing a latent vector based on the original feature vector and the plurality of original attribute values.

3. The method of claim 2 , further comprising:
applying the generator network to the original feature vector to obtain the original image.

4. The method of claim 2 , further comprising:
applying a projection algorithm on the original image to obtain the original feature vector.

5. The method of claim 2 , further comprising:
applying an attribute classifier to the original image to obtain the original attribute values.

6. The method of claim 1 , further comprising:
generating an attribute vector including the target attribute value, wherein the attribute vector has fewer dimensions than the modified feature vector, and wherein the CNF block takes the attribute vector as input.

7. The method of claim 6 , further comprising:
applying an inverse of the mapping network to the original feature vector to obtain a latent vector.

8. The method of claim 6 , wherein:
the mapping network is trained jointly on a plurality of attributes.

9. The method of claim 1 , wherein:
the original image comprises an image of a face, and the target attribute comprises a non-localized attribute including a facial expression, an orientation, an age, a lighting property, a gender, or a hairstyle.

10. The method of claim 1 , further comprising:
receiving an input from a user indicating the target attribute value, wherein the input is received using an image editing application during editing of the original image; and
displaying the modified image to the user via the image editing application in response to the input.

11. A method for image processing, comprising:
identifying an original feature vector representing a plurality of original attributes;
identifying a plurality of original attribute values corresponding to the plurality of original attributes;
computing a latent vector based on the original feature vector and the plurality of original attribute values, wherein the latent vector is computed using an inverse of a mapping network;
identifying one or more target attribute values corresponding to one or more target attributes, wherein the one or more target attributes correspond to a subset of the plurality of original attributes and one or more preserved attribute values correspond to a remaining subset of the plurality of original attributes that do not correspond to the one or more target attributes;
computing a modified feature vector based on the latent vector using the mapping network, wherein the mapping network comprises a continuous normalizing flow (CNF) block that includes a non-linear activation function on the one or more target attribute values and the one or more preserved attribute values; and
generating a modified image based on the modified feature vector using a generator network, wherein the modified image includes the one or more target attributes and the remaining subset of the plurality of original attributes.

12. The method of claim 11 , wherein:
the mapping network is trained jointly based on a plurality of attributes, and wherein the plurality of original attributes correspond to attributes used to train the mapping network.

13. The method of claim 11 , wherein:
the mapping network is configured to enable changing the one or more target attributes while preserving the remaining subset of the plurality of original attributes by correcting for interconnection among the plurality of original attributes in the latent vector based on a non-linear dependency.

14. An apparatus for image processing, comprising:
a mapping network configured to produce a modified feature,
wherein the mapping network comprises a continuous normalizing flow (CNF) block that includes a non-linear activation function and that takes a target attribute value as input, the target attribute value indicating a change to a target attribute of a plurality of semantic attributes of an original image, the target attribute representing a complex set of features of the original image, and
wherein the modified feature vector incorporates the change to the target attribute while holding at least one preserved attribute of the plurality of semantic attributes unchanged; and

a generator network configured to generate a modified image based on the modified feature vector, wherein the modified image includes the change to the target attribute and retains the at least one preserved attribute from the original image.

15. The apparatus of claim 14 , wherein:
the mapping network and the generator network are components of a generative adversarial network (GAN).

16. The apparatus of claim 14 , wherein:
the mapping network and the generator network are components of a variational autoencoder (VAE).

17. The apparatus of claim 14 , wherein:
the mapping network comprises one or more continuous normalizing flow (CNF) blocks.

18. The apparatus of claim 14 , further comprising:
an attribute classifier configured to identify a plurality of attribute values based on an input image.

19. The apparatus of claim 14 , wherein:
the mapping network implements an invertible function from a latent vector space to a feature vector space.

20. The apparatus of claim 14 , wherein:
the mapping network and the generator network are trained using images of human faces.

Assignees

• Adobe Inc
• Univ King Abdullah Sci & Tech

Inventors

• Abdal Rameen
• Wonka Peter
• Mitra Niloy
• Zhu Peihao

Classifications

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title