Extracting gradient features from neural networks

Patent metadata

Field	Value
Publication number	US-2017011280-A1
Application number	US-201514793434-A
Country	US
Kind code	A1
Filing date	Jul 7, 2015
Priority date	Jul 7, 2015
Publication date	Jan 12, 2017
Grant date	—

Abstract

A method for extracting a representation from an image includes inputting an image to a pre-trained neural network. The gradient of a loss function is computed with respect to parameters of the neural network, for the image. A gradient representation is extracted for the image based on the computed gradients, which can be used, for example, for classification or retrieval.

First claim

What is claimed is:

1 . A method for extracting a representation from an image, comprising:
inputting an image to a pre-trained neural network; computing a gradient of a loss function with respect to parameters of the neural network for the image; and extracting a gradient representation of the image based on the computed gradients, wherein at least one of the computing and the extracting is performed with a processor.

2 . The method of claim 1 , wherein the computing of the gradient of the loss function comprises:
computing and extracting a set of forward features from the neural network; computing and extracting a set of backward features from the neural network, the backward features comprising a gradient of the loss with respect to the output of a selected layer of the neural network computed in a backward pass of the neural network; and combining the forward and the backward features to construct a set of gradient features, the gradient features comprising a gradient of the loss with respect to the parameters of a selected layer of the neural network.

3 . The method of claim 1 , wherein the forward and backward features are combined by matrix multiplication.

4 . The method of claim 1 , wherein in a forward pass of the neural network, a prediction vector of values is output for the image which includes a prediction value for each of a set of classes, the computing the gradient of a loss function comprising computing a vector of error values based on a difference between the prediction vector and a standard vector comprising a standard value for each of the classes, and backpropagating the error values through at least one of the layers of the neural network.

5 . The method of claim 1 , wherein the parameters of the neural network comprise parameters of at least one of the fully connected layers of the neural network.

6 . The method of claim 1 , further comprising outputting at least one of:
the vectorial representation of the image, and information based thereon.

7 . The method of claim 6 , wherein the information comprises a computed similarity between the input image and another image based on respective gradient representations of the image and the other image.

8 . The method of claim 7 , wherein the method includes computing the similarity between the two images, comprising:
computing a forward similarity between forward representations of the two images extracted in a forward pass of the neural network, computing a backward similarity between the backward representations of the two images, and computing the similarity by combining the forward and backward similarities.

9 . The method of claim 8 , wherein the combining of the forward and backward similarities is performed by multiplication.

10 . The method of claim 7 , further comprising retrieving a set of similar images based on computed similarity between the input image and each a collection of images.

11 . The method of claim 6 , wherein the gradient representation is classified with a classifier trained on gradient representations of labeled images and the information is based on the classifier classification.

12 . The method of claim 1 wherein the neural network is a convolutional network.

13 . The method of claim 1 , wherein the gradient of the loss function is computed with respect to the weights of at least one of the fully-connected layers according to:

∂
E

∂

W
k

=

x

k
-
1



[

∂
E

∂

y
k

]

T

.

14 . The method of claim 1 , wherein the neural network has been pre-trained to predict labels for an image, the training having been performed with a set of labeled training images.

15 . A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer, causes the computer to perform the method of claim 1 .

16 . A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

17 . A system for extracting a representation from an image, comprising:
memory which stores a pre-trained neural network; a prediction component for predicting labels for an input image using a forward pass of the neural network, and a gradient computation component for computing the gradient of a loss function with respect to parameters of the neural network for the image based on the predicted labels on a backward pass of the neural network; a gradient representation generator for extracting a gradient representation of the image based on the computed gradient; an output component which outputs the gradient representation or information based thereon; and a processor in communication with the memory for implementing the gradient component and prediction component.

18 . The system of claim 17 further comprising at least one of:
a classifier for classifying the input image based on its gradient representation, and
a similarity component for computing a similarity between the input image and another image based on respective gradient representations of the image and the other image.

19 . A method for extracting a representation from an image, comprising:
generating a vector of label predictions for an input image in a forward pass of a pre-trained neural network; computing an error vector based on differences between the vector of label predictions and a standardized prediction vector; in a backward pass of the neural network with the error vector, computing the gradient of a loss function with respect to parameters of the neural network for the image; extracting a gradient representation of the image based on the computed gradients; and outputting the gradient representation or information based thereon, wherein at least one of the computing and the extracting is performed with a processor.

20 . The method of claim 19 , wherein the standardized vector is an equal-valued vector.

Assignees

• Xerox Corp

Inventors

• Soldevila Albert Gordo
• Gaidon Adrien
• Perronnin Florent C

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V30/194

  References adjustable by an adaptive method, e.g. learning · CPC title