Neural Network Patch Aggregation and Statistics

Patent metadata

Field	Value
Publication number	US-2016140408-A1
Application number	US-201414548170-A
Country	US
Kind code	A1
Filing date	Nov 19, 2014
Priority date	Nov 19, 2014
Publication date	May 19, 2016
Grant date	—

Abstract

Neural network patch aggregation and statistical techniques are described. In one or more implementations, patches are generated from an image, e.g., randomly, and used to train a neural network. An aggregation of outputs of patches processed by the neural network may be used to label an image using an image descriptor, such as to label aesthetics of the image, classify the image, and so on. In another example, the patches may be used by the neural network to calculate statistics describing the patches, such as to describe statistics such as minimum, maximum, median, and average of activations of image characteristics of the individual patches. These statistics may also be used to support a variety of functionality, such as to label the image as described above.

First claim

What is claimed is:

1 . A method comprising:
generating a plurality of patches from an image by one or more computing devices; calculating activations of a plurality of image characteristics for each of the plurality of patches using a neural network; sorting the calculated activations according to respective said image characteristics by the one or more computing devices; aggregating the sorted activations for each of the plurality of image characteristics by the one or more computing devices; and labeling the image using an image descriptor based on the aggregated and sorted activations by the one or more computing devices, the image descriptor indicative of a confidence that the image has an image attribute involving the plurality of image characteristics.

2 . A method as described in claim 1 , wherein the generating is performed such that the plurality of patches are selected randomly from the image in an order-less manner.

3 . A method as described in claim 1 , wherein the plurality of patches are taken from the image without down-sampling the image.

4 . A method as described in claim 1 , wherein the aggregating includes ranking the calculated activations, one to another, for the image characteristic.

5 . A method as described in claim 1 , wherein the sorting includes learning fixed weights for respective said patches and the aggregating is performed using the fixed weights.

6 . A method as described in claim 1 , wherein:
the calculated activations for a respective said path are included in a vector that has a plurality of dimensions, each of the dimensions corresponding to respective said image characteristic; and the sorting includes sorting the calculated activations on individual said dimensions of the plurality of vectors.

7 . A method as described in claim 6 , wherein the aggregating includes concatenating the sorted output of the calculated activations and the labeling is based on a patch aggregation weight learned from the concatenating.

8 . A method as described in claim 7 , wherein the patch aggregation weight is learned using a fully connected layer of the neural network, the sorting including recording an order of individual outputs of the calculated activations, back propagation is performed to assign a corresponding gradient descent to respective said patches based on the recorded order, and the patch aggregation weights are shared with other said patches as part of computation of the patch aggregation weights.

9 . A method as described in claim 1 , wherein the neural network is a single-column patch neural network (SPNN) that is configured to perform the calculating of the activations for each of the plurality of patches.

10 . A method as described in claim 1 , wherein the image attribute involves face quality classification, image quality classification, or image aesthetics.

11 . A method comprising:
generating a plurality of patches from an image by one or more computing devices; calculating activations of a plurality of image characteristics for each of the plurality of patches using a neural network; sorting the calculated activations according to respective said image characteristics by the one or more computing devices; calculating one or more statistics using the sorted activations for each of the plurality of image characteristics by the one or more computing devices; and labeling the image by the one or more computing devices using an image descriptor based on the calculated statistics, the image descriptor indicative of a confidence that the image has an image attribute involving the plurality of image characteristics.

12 . A method as described in claim 11 , wherein the statistics include a minimum, maximum, median, or average of the calculated activations.

13 . A method as described in claim 11 , wherein the calculating of the statistics is performed as part of an iterative network training framework in which an error generated based on the one or more statistics contributes to training of the neural network.

14 . A method as described in claim 11 , wherein the generating is performed such that the plurality of patches are selected randomly from the image in an order-less manner.

15 . A method as described in claim 11 , further comprising concatenating output vectors that include the calculated activations and training fully-connected layers of the neural network for classification using the calculated statistics as part of the labeling.

16 . A system comprising:
a patch generation module implemented at least partially in hardware, the patch generation module configured to generate a plurality of patches through random selection from an image; a neural network module implemented at least partially in hardware, the neural network module configured to generate a vector using a neural network for each of the plurality of patches having activations in dimensions corresponding to respective ones of a plurality of image characteristics; a sorting and aggregation module implemented at least partially in hardware, the sorting and aggregation module configured to sort the calculated activations according to respective said image characteristics and aggregate the sorted activations for each of the plurality of image characteristics; and a labeling module implemented at least partially in hardware, the labeling module configured to label the image using an image descriptor based on the aggregated and sorted activations, the image descriptor indicative of a confidence that the image has an image attribute involving the plurality of image characteristics.

17 . A system as described in claim 16 , wherein the sorting includes learning fixed weights for respective said patches and the aggregating is performed using the fixed weights.

18 . A system as described in claim 16 , wherein:
the calculated activations for a respective said path are included in a vector that has a plurality of dimensions, each of the dimensions corresponding to respective said image characteristic; and the sorting includes sorting the calculated activations on individual said dimensions of the plurality of vectors.

19 . A system as described in claim 18 , wherein the aggregating includes concatenating the sorted output of the calculated activations and the labeling is based on a patch aggregation weight learned from the concatenating.

20 . A system as described in claim 19 , wherein the patch aggregation weight is learned using a fully connected layer of the neural network, the sorting including recording an order of individual outputs of the calculated activations, back propagation is performed to assign a corresponding gradient descent to respective said patches based on the recorded order, and the patch aggregation weights are shared with other said patches as part of computation of the patch aggregation weights.

Assignees

• Adobe Systems Inc

Inventors

• Shen Xiaohui
• Lu Xin
• Lin Zhe
• Mech Radomir

Classifications

• G06V10/464

  using a plurality of salient features, e.g. bag-of-words [BoW] representations · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/048

  Activation functions · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title