Attention-driven image manipulation

Patent metadata

Field	Value
Publication number	US-11138693-B2
Application number	US-202016752030-A
Country	US
Kind code	B2
Filing date	Jan 24, 2020
Priority date	Jan 24, 2020
Publication date	Oct 5, 2021
Grant date	Oct 5, 2021

Abstract

Techniques of adjusting the salience of an image include generating values of photographic development parameters for a foreground and background of an image to adjust the salience of the image in the foreground. These parameters are global in nature over the image rather than local. Moreover, the optimization of the salience over such sets of global parameters is provided through two sets of these parameters by an encoder: one set corresponding to the foreground, in which the salience is to be either increased or decreased, and the other set corresponding to the background. Once the set of development parameters corresponding to the foreground region and the set of development parameters corresponding to the background region have been determined, a decoder generates an adjusted image with an increased salience based on these sets of development parameters.

First claim

What is claimed is:

1. A method, comprising:
receiving (i) initial image data representing an initial image having a plurality of pixels and (ii) object mask data, the object mask data indicating a foreground region of the plurality of pixels, the object mask data further indicating a background region of the plurality of pixels;
generating (i) foreground image parameter values corresponding to the foreground region and (ii) background image parameter values corresponding to the background region, the foreground image parameter values and the background image parameter values being based on the initial image data and the object mask data, wherein generating the foreground image parameter values and the background image parameter values includes inputting the initial image data and the object mask data into an encoder of a machine learning engine, the encoder including at least one neural network, and the encoder being configured to generate a saliency map of the initial image;
generating adjusted image data representing an adjusted image, the adjusted image being based on the initial image data, the foreground image parameter values, and the background image parameter values, the adjusted image having different salience values in the foreground region than the initial image; and
outputting the adjusted image.

2. The method as in claim 1 ,
wherein the encoder includes a plurality of neural networks;
wherein generating the foreground image parameter values and the background image parameter values includes outputting the foreground image parameter values and the background image parameter values by a first neural network of the plurality of neural networks, and
wherein generating the adjusted image data includes inputting the initial image data, the foreground image parameter values, and the background image parameter values into a decoder of the machine learning engine, the decoder including a fixed set of functions, each of the fixed set of functions being based on the foreground image parameter values and the background image parameter values and being applied to the initial image data to produce the adjusted image values.

3. The method as in claim 2 , wherein generating the foreground image parameter values and the background image parameter values further includes:
inputting the initial image data into a second neural network of the plurality of neural networks to produce a first output indicative of semantic content of the initial image;
generating initial salience data representing a saliency map of the initial image;
inputting the object mask data and the initial salience data into a third neural network of the plurality of neural networks to produce a second output indicative of a difference between the saliency map of the foreground region of the initial image and a saliency map of the foreground region of the adjusted image;
combining the first output and the second output to produce a third output; and
inputting the third output into the neural network of the plurality of neural networks to produce the foreground image parameter values and the background image parameter values upon being input.

4. The method as in claim 3 , wherein the first output includes a vector of a specified size, each element of the vector including a representation of a feature extracted from the initial image data.

5. The method as in claim 4 , wherein the representation of a feature included in an element of the vector has a size based on a layer of the second neural network from which the representation of the feature is extracted.

6. The method as in claim 3 , further comprising performing a global average pooling operation on the first output.

7. The method as in claim 3 , further comprising performing a global average pooling operation on the second output.

8. A system configured to adjust a salience of selected regions of images, the system comprising:
at least one memory including instructions; and
at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the at least one processor to implement an application, the application comprising:
a graphical user interface (GUI) configured to indicate (i) initial image data representing an initial image having a plurality of pixels and (ii) object mask data indicating a foreground region of the plurality of pixels and a background region of the plurality of pixels, the GUI being displayed in a display device;
a machine learning engine including:
an encoder configured to generate (i) foreground image parameter values corresponding to the foreground region and (ii) background image parameter values corresponding to the background region based on the initial image data and the object mask data;
a decoder configured to generate adjusted image data representing an adjusted image, the adjusted image being based on the initial image data, the foreground image parameter values, and the background image parameter values, the adjusted image having increased salience values in the foreground region; and

a display engine configured to display the adjusted image in an output window of the GUI.

9. The system as in claim 8 , wherein the adjusted image data is produced from the initial image data and the object mask data based on a machine learning engine, the machine learning engine including an encoder and a decoder,
the encoder including a plurality of neural networks, a first neural network of the plurality of neural networks outputting the foreground image parameter values and the background image parameter values,
the decoder including a fixed set of functions, each of the fixed set of functions being based on the foreground image parameter values and the background image parameter values and being applied to the initial image data to produce the adjusted image values,
wherein the instructions that, when executed, cause the at least one processor to generate the foreground image parameter values and the background image parameter values further cause the at least one processor to input the initial image data and the object mask data into the encode, and
wherein the instructions that, when executed, cause the at least one processor to generate the adjusted image data further cause the at least one processor to input the initial image data, the foreground image parameter values, and the background image parameter values into the decoder.

10. The system as in claim 9 , wherein the first neural network includes fully connected layers.

11. The system as in claim 9 , wherein the instructions that, when executed, cause the at least one processor to generate the adjusted image data further cause the at least one processor to:
generate an intermediate foreground image based on (i) the initial image in the foreground region and the foreground image parameter values and (ii) the initial image in the background region and the background image parameter values; and
generate an intermediate background image based on (i) the initial image in the foreground region and the foreground image parameter values and (ii) the initial image in the background region and the background image parameter values.

12. The system as in claim 11 , wherein the instructions that, when executed, cause the at least one processor to generate the intermediate foreground image further cause the at least one processor to:
apply the decoder to the initial image in the foreground region and the foreground image parameter values to produce a first output;
apply the decoder to the initial image in the background region and the background image parameter values to produ

Assignees

• Adobe Inc

Inventors

• Mejjati Youssef Alami
• Bylinskii Zoya
• Shechtman Elya

Classifications

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/045

  Combinations of networks · CPC title