Tagging similar images using neural network

Patent metadata

Field	Value
Publication number	US-9928449-B2
Application number	US-201715494891-A
Country	US
Kind code	B2
Filing date	Apr 24, 2017
Priority date	Feb 5, 2016
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

An approach is provided in which a knowledge manager selects an extraction layer from a convolutional neural network that was trained on an initial set of images. The knowledge manager processes subsequent images obtained from crawling a computer network that includes extracting image feature sets of the subsequent images from the selected extraction layer and generating tags from metadata associated with the subsequent images. In turn, the knowledge manager receives a new image, extracts a new image feature set from the selected extraction layer, and assigns one or more of the tags to the new image based upon evaluating the new image feature set to the image features sets of the subsequent images.

First claim

The invention claimed is:

1. A method implemented by an information handling system that includes a memory and a processor, the method comprising:
training a classifier on a first set of images, wherein the classifier comprises a convolutional neural network that includes a plurality of layers;
processing a second set of images using the trained classifier, wherein the processing comprises:
extracting a plurality of feature sets from a selected layer included in the plurality of layers, wherein the plurality of layers includes a set of convolutional layers and a subsequent set of fully connected non-convolutional layers, and wherein the selected layer is a first layer in the subsequent set of fully-connected non-convolutional layers; and
generating a plurality of tags based upon metadata corresponding to the second set of images; and

assigning at least one of the plurality of tags to a new image based upon evaluating a new image feature set against one or more of the plurality of feature sets, wherein the new image feature set is extracted from the selected layer during processing of the new image.

2. The method of claim 1 wherein the selected layer is not a last layer in the plurality of layers.

3. The method of claim 1 further comprising:
grouping the plurality of feature sets into a plurality of clusters based upon performing a plurality of nearest neighbor searches on the plurality of feature sets.

4. The method of claim 1 further comprising:
performing a nearest neighbors search on the new image feature set, resulting in a selection of a subset of the plurality of tags; and
voting on the subset of the plurality of tags to arrive at the assigned at least one tag.

5. The method of claim 1 wherein the first set of images includes a first set of object types that are different from a second set of object types included in the second set of images.

6. The method of claim 1 further comprising searching the Internet to obtain the second set of images.

7. An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors; and
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of:
training a classifier on a first set of images, wherein the classifier comprises a convolutional neural network that includes a plurality of layers;
processing a second set of images using the trained classifier, wherein the processing comprises:
extracting a plurality of feature sets from a selected layer included in the plurality of layers, wherein the plurality of layers includes a set of convolutional layers and a subsequent set of fully connected non-convolutional layers, and wherein the selected layer is a first layer in the subsequent set of fully-connected non-convolutional layers; and
generating a plurality of tags based upon metadata corresponding to the second set of images; and

assigning at least one of the plurality of tags to a new image based upon evaluating a new image feature set against one or more of the plurality of feature sets, wherein the new image feature set is extracted from the selected layer during processing of the new image.

8. The information handling system of claim 7 wherein the selected layer is not a last layer in the plurality of layers.

9. The information handling system of claim 7 wherein at least one of the one or more processors perform additional actions comprising:
grouping the plurality of feature sets into a plurality of clusters based upon performing a plurality of nearest neighbor searches on the plurality of feature sets.

10. The information handling system of claim 7 wherein at least one of the one or more processors perform additional actions comprising:
performing a nearest neighbors search on the new image feature set, resulting in a selection of a subset of the plurality of tags; and
voting on the subset of the plurality of tags to arrive at the assigned at least one tag.

11. The information handling system of claim 7 wherein the first set of images includes a first set of object types that are different from a second set of object types included in the second set of images.

12. The information handling system of claim 7 wherein at least one of the one or more processors perform additional actions comprising:
searching the Internet to obtain the second set of images.

13. A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:
training a classifier on a first set of images, wherein the classifier comprises a convolutional neural network that includes a plurality of layers;
processing a second set of images using the trained classifier, wherein the processing comprises:
extracting a plurality of feature sets from a selected layer included in the plurality of layers, wherein the plurality of layers includes a set of convolutional layers and a subsequent set of fully connected non-convolutional layers, and wherein the selected layer is a first layer in the subsequent set of fully-connected non-convolutional layers; and
generating a plurality of tags based upon metadata corresponding to the second set of images; and

assigning at least one of the plurality of tags to a new image based upon evaluating a new image feature set against one or more of the plurality of feature sets, wherein the new image feature set is extracted from the selected layer during processing of the new image.

14. The computer program product of claim 13 wherein the selected layer is not a last layer in the plurality of layers.

15. The computer program product of claim 13 wherein the information handling system performs additional actions comprising:
grouping the plurality of feature sets into a plurality of clusters based upon performing a plurality of nearest neighbor searches on the plurality of feature sets.

16. The computer program product of claim 13 wherein the information handling system performs additional actions comprising:
performing a nearest neighbors search on the new image feature set, resulting in a selection of a subset of the plurality of tags; and
voting on the subset of the plurality of tags to arrive at the assigned at least one tag.

17. The computer program product of claim 13 wherein the first set of images includes a first set of object types that are different from a second set of object types included in the second set of images.

18. The computer program product of claim 13 wherein the information handling system performs additional actions comprising:
searching the Internet to obtain the second set of images.

Assignees

• IBM

Inventors

• Chavez Aaron J
• Harper Devin R
• Lineback Nicholas A
• Turner Elliot B

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06F18/24147

  Distances to closest patterns, e.g. nearest neighbour classification · CPC title

• G06N3/045

  Combinations of networks · CPC title