Image ranking based on attribute correlation

Patent metadata

Field	Value
Publication number	US-9262445-B2
Application number	US-201414516943-A
Country	US
Kind code	B2
Filing date	Oct 17, 2014
Priority date	Jun 3, 2011
Publication date	Feb 16, 2016
Grant date	Feb 16, 2016

Abstract

Images are retrieved and ranked according to relevance to attributes of a multi-attribute query through training image attribute detectors for different attributes annotated in a training dataset. Pair-wise correlations are learned between pairs of the annotated attributes from the training dataset of images. Image datasets may are searched via the trained attribute detectors for images comprising attributes in a multi-attribute query. The retrieved images are ranked as a function of comprising attributes that are not within the query subset plurality of attributes but are paired to one of the query subset plurality of attributes by the pair-wise correlations, wherein the ranking is an order of likelihood that the different ones of the attributes will appear in an image with the paired one of the query subset plurality of attributes.

First claim

What is claimed is:

1. A method for retrieving and ranking multi-attribute query results according to relevance to attributes of a multi-attribute query, the method comprising:
training each of a plurality of image attribute detectors for one each of a plurality of different attributes that are annotated in a training dataset of images of people;
learning via a processor a plurality of pair-wise correlations between each pair of the plurality of annotated attributes from the training dataset of images;
searching via the trained attribute detectors an input image dataset for images comprising at least one of a multi-attribute query subset plurality of the annotated attributes;
retrieving a plurality of images from the searching of the input image dataset that each comprise at least one of the query subset plurality of attributes; and
ranking the retrieved plurality of images as a function of comprising different ones of the attributes that are not within the query subset plurality of attributes but are paired to the at least one of the query subset plurality of attributes by the pair-wise correlations, wherein the ranking is an order of likelihood that the different ones of the attributes will appear in an image with the paired at least one of the query subset plurality of attributes.

2. The method of claim 1 , further comprising:
integrating computer-readable program code into a computer system comprising the processor, a computer readable memory in circuit communication with the processor, and a computer readable storage medium in circuit communication with the processor; and
wherein the processor executes program code instructions stored on the computer-readable storage medium via the computer readable memory and thereby performs the steps of training each of the plurality of image attribute detectors, learning the plurality of pair-wise correlations between each pair of the plurality of annotated attributes from the training dataset of images, searching via the trained attribute detectors the input image dataset for images comprising at least one of a multi-attribute query subset plurality of the annotated attributes, retrieving the plurality of images from the searching of the input image dataset that each comprise at least one of the query subset plurality of attributes, and ranking the retrieved plurality of images as a function of comprising different ones of the attributes that are not within the query subset plurality of attributes but are paired to the at least one of the query subset plurality of attributes by the pair-wise correlations.

3. The method of claim 1 , wherein a first of the annotated attributes is more heavily weighted than a second of the annotated attributes; and
wherein the ranking the retrieved plurality of images further comprises ranking a one of the results with the more heavily weighted first attribute higher than another of the results that has the second attribute and a same total number of the attributes that are also within the query subset plurality of attributes.

4. The method of claim 3 , wherein the learning the plurality of pair-wise correlations between each pair of the plurality of annotated attributes further comprises:
reverse learning a mapping of a set of labels of the annotated attributes to the images in the training dataset of images to predict respective sets of the training dataset images that each contain one of the annotated attribute labels.

5. The method of claim 4 , wherein the retrieving the plurality of images from the searching of the input image dataset that each comprise at least one of the query subset plurality of attributes further comprises:
predicting the retrieved plurality of images by maximizing weighted feature vectors extracted by each of the trained image attribute detectors as a function of a component modeling an appearance of the attribute of the each of the trained image attribute detectors, and a component modeling a dependency between the attribute of the each of the attributes of the trained image attribute detectors to another one of the annotated attributes in the training dataset of images that is not in the query subset plurality of attributes.

6. The method of claim 5 , wherein the learning the pair-wise correlations is a max-margin training.

7. The method of claim 6 , wherein the predicting the retrieved plurality of images by maximizing the weighted feature vectors extracted by each of the trained image attribute detectors further comprises:
employing a complex loss function to more heavily penalize one of the weighted feature vector outputs that deviates more from a correct output measured based on an optimized performance metric than another of the weighted feature vector outputs having a lesser deviation from the correct output measured based on the optimized performance metric.

8. The method of claim 7 , wherein the max-margin training further comprises:
generating a plurality of constraints; and
iteratively adding most violated constraints of the generating a plurality of constraints to the optimized performance metric.

9. A system, comprising:
a processing unit, a computer readable memory and a computer readable storage medium;
first program instructions to train each of a plurality of image attribute detectors for one each of a plurality of different attributes that are annotated in a training dataset of images of people;
second program instructions to learn a plurality of pair-wise correlations between each pair of the plurality of annotated attributes from the training dataset of images;
third program instructions to search via the trained attribute detectors an input image dataset for images comprising at least one of a multi-attribute query subset plurality of the annotated attributes; and
fourth program instructions to retrieve a plurality of images from the searching of the input image dataset that each comprise at least one of the query subset plurality of attributes, the fourth program instructions further to rank the retrieved plurality of images as a function of comprising different ones of the attributes that are not within the query subset plurality of attributes but are paired to the at least one of the query subset plurality of attributes by the pair-wise correlations, wherein the ranking is an order of likelihood that the different ones of the attributes will appear in an image with the paired at least one of the query subset plurality of attributes; and
wherein the first, second, third and fourth program instructions are stored on the computer readable storage medium for execution by the processing unit via the computer readable memory.

10. The system of claim 9 , wherein a first of the annotated attributes is more heavily weighted than a second of the annotated attributes; and
wherein the fourth program instructions are further to rank the retrieved plurality of images by ranking a one of the results with the more heavily weighted first attribute higher than another of the results that has the second attribute and a same total number of the attributes that are also within the query subset plurality of attributes.

11. The system of claim 10 , wherein the second program instructions are further to learn the plurality of pair-wise correlations between each pair of the plurality of annotated attributes by reverse learning a mapping of a set of labels of the annotated attributes to the images in the training dataset of images to predict respective sets of the training dataset images that each contain one of the annotated attribute labels.

12. The system of claim 11 , wherein the fourth program instructions are further to retrieve the plurality of images fro

Assignees

• IBM

Inventors

• Datta Ankur
• Feris Rogerio S
• Pankanti Sharathchandra U
• Siddiquie Behjat

Classifications

• G06V30/19167

  Active pattern learning · CPC title

• G06F16/532

  Query formulation, e.g. graphical querying · CPC title

• G06F16/24578

  using ranking · CPC title

• G06N20/10

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06F16/5838 Primary

  using colour · CPC title