Attribute decorrelation techniques for image editing

Patent metadata

Field	Value
Publication number	US-11875221-B2
Application number	US-202117468476-A
Country	US
Kind code	B2
Filing date	Sep 7, 2021
Priority date	Oct 16, 2020
Publication date	Jan 16, 2024
Grant date	Jan 16, 2024

Abstract

Systems and methods generate a filtering function for editing an image with reduced attribute correlation. An image editing system groups training data into bins according to a distribution of a target attribute. For each bin, the system samples a subset of the training data based on a pre-determined target distribution of a set of additional attributes in the training data. The system identifies a direction in the sampled training data corresponding to the distribution of the target attribute to generate a filtering vector for modifying the target attribute in an input image, obtains a latent space representation of an input image, applies the filtering vector to the latent space representation of the input image to generate a filtered latent space representation of the input image, and provides the filtered latent space representation as input to a neural network to generate an output image with a modification to the target attribute.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
obtaining training data;
grouping the training data into a plurality of bins according to a distribution of a target attribute in the training data;
for each bin, of the plurality of bins, sampling a subset of the training data in the bin to generate sampled training data based on a pre-determined target distribution of a set of additional attributes in the training data;
identifying a direction in the sampled training data corresponding to the distribution of the target attribute to generate a filtering vector for modifying the target attribute in an input image;
obtaining a latent space representation of an input image;
applying the filtering vector to the latent space representation of the input image to generate a filtered latent space representation of the input image; and
providing the filtered latent space representation as input to a neural network to generate an output image with a modification to the target attribute.

2. The method of claim 1 , wherein sampling the subset of the training data comprises: determining a marginal distribution of each additional attribute of the set of additional attributes;
setting each marginal distribution to a value corresponding to the pre-determined target distribution to control correlations between the attributes; and
taking a union of the marginal distributions of a set of possible combinations of attribute values.

3. The method of claim 1 , wherein the neural network is a generator comprising a plurality of layers, and wherein providing the filtered latent space representation as input to the neural network further comprises:
selecting a subset of the layers associated with the target attribute; and
feeding the filtered latent space representation to the selected subset of the layers.

4. The method of claim 1 , wherein:
the target attribute is modified to a plurality of states without modification to the additional attributes.

5. The method of claim 1 , further comprising:
causing display of a user interface comprising a plurality of interactive components configured to control a respective plurality of attributes including the target attribute; and
receiving input specifying the modification to the target attribute, wherein the filtering vector is generated and applied based on the specified modification.

6. The method of claim 1 , wherein:
the target attribute is a non-binary attribute; and
the training data is divided into three or more bins, each ban spanning a range of attribute values.

7. The method of claim 1 , wherein obtaining the latent space representation of the input image comprises generating the latent space representation of the input image by applying a machine learning model to the input image.

8. A computing system comprising:
a memory;
a processor;
a non-transitory computer-readable medium comprising instructions which, when executed by the processor, perform the steps of:
grouping training data into a plurality of bins according to a distribution of a target attribute in the training data;
for each bin, of the plurality of bins, sampling a subset of the training data in the bin to generate sampled training data based on a pre-determined target distribution of a set of additional attributes in the training data;
identifying a direction in the sampled training data corresponding to the distribution of the target attribute to generate a filtering vector for modifying the target attribute in an input image;
applying the filtering vector to a latent space representation of the input image to generate a filtered latent space representation of the input image; and
providing the filtered latent space representation as input to a neural network to generate an output image with a modification to the target attribute.

9. The computing system of claim 8 , wherein sampling the subset of the training data comprises:
determining a marginal distribution of each additional attribute of the set of additional attributes;
setting each marginal distribution to a value corresponding to the pre-determined target distribution to control correlations between the attributes; and
taking a union of the marginal distributions of a set of possible combinations of attribute values.

10. The computing system of claim 8 , wherein:
the target attribute is modified to a plurality of states without modification to the additional attributes.

11. The computing system of claim 8 , wherein the neural network is a generator comprising a plurality of layers, and wherein providing the filtered latent space representation as input to the neural network further comprises:
selecting a subset of the layers associated with the target attribute; and
feeding the filtered latent space representation to the selected subset of the layers.

12. The computing system of claim 8 , the steps further comprising:
causing display of a user interface comprising a plurality of interactive components configured to control a respective plurality of attributes including the target attribute; and
receiving input specifying the modification to the target attribute, wherein the filtering vector is generated and applied based on the specified modification.

13. The computing system of claim 8 , wherein:
the target attribute is a non-binary attribute; and
the training data is divided into three or more bins, each ban spanning a range of attribute values.

14. The computing system of claim 8 , wherein obtaining the latent space representation of the input image comprises:
generating the latent space representation of the input image by applying a machine learning model to the input image.

15. A non-transitory computer-readable medium having instructions stored thereon, the instructions executable by a processing device to perform operations comprising:
obtaining training data;
grouping the training data into a plurality of bins according to a distribution of a target attribute in the training data;
for each bin, of the plurality of bins, sampling a subset of the training data in the bin to generate sampled training data based on a pre-determined target distribution of a set of additional attributes in the training data;
identifying a direction in the sampled training data corresponding to the distribution of the target attribute to generate a filtering vector for modifying the target attribute in an input image;
obtaining a latent space representation of an input image;
applying the filtering vector to a latent space representation of an input image to generate a filtered latent space representation of the input image; and
providing the filtered latent space representation as input to a neural network to generate an output image with a modification to the target attribute.

16. The non-transitory computer-readable medium of claim 15 , wherein the neural network is a generator comprising a plurality of layers, and wherein providing the filtered latent space representation as input to the neural network further comprises:
selecting a subset of the layers associated with the target attribute; and
feeding the filtered latent space representation to the selected subset of the layers.

17. The non-transitory computer-readable medium of claim 15 , wherein:
the target attribute is modified to a plurality of states without modification to the additional attributes.

18. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
causing display of a user interface comprising a plurality of interac

Assignees

• Adobe Inc

Inventors

• Lin Wei-An
• Faieta Baldo
• Smith Cameron
• Shechtman Elya
• Lu Jingwan
• Zhu Jun-Yan
• Mitra Niloy
• Kalarot Ratheesh
• Zhang Richard
• Ghadar Shabnam
• Shu Zhixin

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/094 Primary

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title