Method for determining similarity of objects represented in images

Patent metadata

Field	Value
Publication number	US-9436895-B1
Application number	US-201514678102-A
Country	US
Kind code	B1
Filing date	Apr 3, 2015
Priority date	Apr 3, 2015
Publication date	Sep 6, 2016
Grant date	Sep 6, 2016

Abstract

A method re-identifies objects in a pair of images by applying a convolutional neural network (CNN). Each layer in the network operates on an output of a previous layer. The layers include a first convolutional layer and a first max pooling layer to determine a feature map, a cross-input neighborhood differences layer to produce neighborhood difference maps, a patch summary layer to produce patch summary feature maps, a first fully connected layer to produce a feature vector representing higher order relationships in the patch summary feature maps, a second fully connected layer to produce two scores representing positive pair and negative pair classes, and a softmax layer to produce positive pair and negative pair probabilities. Then, the positive pair probability is output to signal whether the two images represent the same object or not.

First claim

We claim:

1. A method for re-identification of objects, comprising steps:
obtaining a pair of images, wherein each image represents an object;
applying a convolutional neural network (CNN) to the pair of images, wherein the CNN comprises:
a first convolutional layer;
a first max pooling layer, which follows the first convolutional layer, wherein the first convolutional layer and the first max pooling layer are applied to each image separately to determine a feature map for each image;
a cross-input neighborhood differences layer, which is applied to the feature maps to produce neighborhood difference maps;
a patch summary layer, which is applied to the neighborhood difference maps to produce patch summary feature maps;
a first fully connected layer, which is applied to the patch summary feature maps to produce a feature vector representing higher order relationships in the patch summary feature maps;
a second fully connected layer, which is applied to the feature vector representing higher order relationships to produce two scores representing positive pair and negative pair classes;
a softmax layer, which is applied to the two scores to produce positive pair and negative pair probabilities; and
outputting the positive pair probability to signal whether the two images represent the same object or not, wherein the steps are performed in a processor.

2. The method of claim 1 , wherein the objects are persons, and the method is for person re-identification.

3. The method of claim 1 , wherein the network does not include the second fully connected layer nor the softmax layer, wherein the method further comprises using the feature vector representing higher order relationships as input to a classifier, and wherein the outputting comprises outputting a result of the classifier.

4. The method of claim 3 , wherein the classifier is a linear support vector machine (SVM) classifier.

5. The method of claim 3 , wherein the feature vector representing higher order relationships is passed through a rectified linear unit (ReLu) before being used as input to a classifier.

6. The method of claim 1 , wherein a positive pair is defined as two images of the same object, and a negative pair is defined as two images of different objects.

7. The method of claim 1 , wherein positive pair refers to two images from the same object class, negative pair refers to two images from different object classes, and wherein, instead of signaling whether the two images represent the same object or not, the outputting signals whether the two images represent the same object class or not.

8. The method of claim 1 , further comprising:
obtaining the pair of images by a camera.

9. The method of claim 1 , wherein the CNN further comprises:
a second convolutional layer; and
a second max pooling layer, which follows the second convolutional layer, wherein the second convolutional layer takes as input the feature maps determined by the first max pooling layer, the second max pooling layer outputs refined feature maps, and the cross-input neighborhood differences layer is applied to the refined feature maps to produce neighborhood difference maps.

10. The method of claim 9 , wherein the first and second convolutional layers are tied by weights shared across the pair of images, to ensure that identical filters are applied to each image.

11. The method of claim 1 , wherein the CNN further comprises:
a third convolutional layer; and
a third max pooling layer, which follows the third convolutional layer, wherein the third convolutional layer is applied to the patch summary feature maps, the third max pooling layer outputs high-level feature maps, and the first fully connected layer is applied to the high-level feature maps to produce the feature vector representing higher order relationships.

12. The method of claim 1 , wherein the first convolutional layer is tied by weights shared across the pair of images, to ensure that identical filters are applied to each image.

13. The method of claim 1 , wherein the network is trained with example positive pairs and negative pairs.

14. The method of claim 1 , wherein the cross-input neighborhood differences layer accumulates differences in feature values around a neighborhood of each feature location across the two images.

15. The method of claim 1 , wherein the neighborhood difference maps are passed through a rectified linear unit (ReLu) before being used as input to the following layer.

16. The method of claim 1 , wherein the patch summary feature maps are passed through a rectified linear unit (ReLu) before being used as input to the following layer.

17. The method of claim 1 , wherein the feature vector representing higher order relationships is passed through a rectified linear unit (ReLu) before being used as input to the following layer.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Jones Michael
• Marks Tim
• Ahmed Ejaz

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214 Primary

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/2414

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title