Image ranking based on attribute correlation

Patent metadata

Field	Value
Publication number	US-2016124996-A1
Application number	US-201614994501-A
Country	US
Kind code	A1
Filing date	Jan 13, 2016
Priority date	Jun 3, 2011
Publication date	May 5, 2016
Grant date	—

Abstract

Images are retrieved and ranked according to relevance to attributes of a multi-attribute query through training image attribute detectors for different attributes annotated in a training dataset. Pair-wise correlations are learned between pairs of the annotated attributes from the training dataset of images. Image datasets may are searched via the trained attribute detectors for images comprising attributes in a multi-attribute query. The retrieved images are ranked as a function of comprising attributes that are not within the query subset plurality of attributes but are paired to one of the query subset plurality of attributes by the pair-wise correlations, wherein the ranking is an order of likelihood that the different ones of the attributes will appear in an image with the paired one of the query subset plurality of attributes.

First claim

What is claimed is:

1 . A computer-implemented method for retrieving and ranking multi-attribute query results according to relevance to attributes of a multi-attribute query, the method comprising executing on a processor the steps of:
extracting feature vectors from subset portions of image fields of each of a plurality of facial images within a training dataset, wherein each of the facial images comprise different annotated attributes; generating a plurality of different image attribute detectors, one for one each of the annotated attributes, by concatenating the extracted feature vectors for each of the facial images; learning a plurality of pair-wise correlations between each pair of the plurality of annotated attributes from the training dataset of images; retrieving a plurality of images from an input image dataset that each comprise feature vectors that fit at least one of the attribute detectors of a multi-attribute query subset plurality of the annotated attributes; and ranking the retrieved plurality of images as a function of comprising different ones of the attributes that are not within a query subset plurality of attributes but are paired to the at least one of the query subset plurality of attributes by the pair-wise correlations, wherein the ranking is an order of likelihood that the different ones of the attributes will appear in an image with the paired at least one of the query subset plurality of attributes.

2 . The method of claim 1 , further comprising:
integrating computer-readable program code into a computer system comprising the processor, a computer readable memory in circuit communication with the processor, and a computer readable storage medium in circuit communication with the processor; and wherein the processor executes program code instructions stored on the computer-readable storage medium via the computer readable memory and thereby performs the steps of extracting the feature vectors from the subset portions of the image fields of the plurality of facial images within the training dataset, generating the plurality of different image attribute detectors, learning the plurality of pair-wise correlations between each pair of the plurality of annotated attributes from the training dataset of images, retrieving the plurality of images that each comprise the feature vectors that fit at least one of the attribute detectors of the multi-attribute query subset plurality of the annotated attributes, and ranking the retrieved plurality of images as the function of comprising different ones of the attributes that are not within the query subset plurality of attributes but are paired to the at least one of the query subset plurality of attributes by the pair-wise correlations.

3 . The method of claim 1 , wherein the subset portions of the image fields comprise one of:
each of an array of two-dimensional grid portions, wherein each grid portion spans a different two-dimensional portion of an entirety of the facial images; and a subset of the grid portions that comprises less than a totality of the array of grid portions.

4 . The method of claim 3 , wherein the subset of the grid portions comprises:
a central grid portion that is surrounded by outer ones of the array of grid portions; a vertical column of a plurality of the grid portions; or the horizontal row of a plurality of the grid portions.

5 . The method of claim 4 , further comprising:
selecting a one of the central grid portion, the vertical column of grid portions and the horizontal row of grid portions that is most likely to include at least one of the attribute detectors of a multi-attribute query subset plurality of the annotated attributes as the subset portions of the image fields from which the feature vectors are extracted.

6 . The method of claim 4 , wherein the learning the pair-wise correlations is a max-margin training.

7 . The method of claim 6 , wherein the predicting the retrieved plurality of images by maximizing the weighted feature vectors extracted by each of the trained image attribute detectors further comprises:
employing a complex loss function to more heavily penalize one of the weighted feature vector outputs that deviates more from a correct output measured based on an optimized performance metric than another of the weighted feature vector outputs having a lesser deviation from the correct output measured based on the optimized performance metric.

8 . The method of claim 7 , wherein the max-margin training further comprises:
generating a plurality of constraints; and iteratively adding most violated constraints of the generating a plurality of constraints to the optimized performance metric.

9 . A system, comprising:
a processor; a computer readable memory in circuit communication with the processor; and a computer readable storage medium in circuit communication with the processor; wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby: extracts feature vectors from subset portions of image fields of each of a plurality of facial images within a training dataset, wherein each of the facial images comprise different annotated attributes; generates a plurality of different image attribute detectors, one for one each of the annotated attributes, by concatenating the extracted feature vectors for each of the facial images; learns a plurality of pair-wise correlations between each pair of the plurality of annotated attributes from the training dataset of images; retrieves a plurality of images from an input image dataset that each comprise feature vectors that fit at least one of the attribute detectors of a multi-attribute query subset plurality of the annotated attributes; and ranks the retrieved plurality of images as a function of comprising different ones of the attributes that are not within a query subset plurality of attributes but are paired to the at least one of the query subset plurality of attributes by the pair-wise correlations, wherein the ranking is an order of likelihood that the different ones of the attributes will appear in an image with the paired at least one of the query subset plurality of attributes.

10 . The system of claim 9 , wherein the subset portions of the image fields comprise one of:
each of an array of two-dimensional grid portions, wherein each grid portion spans a different two-dimensional portion of an entirety of the facial images; and a subset of the grid portions that comprises less than a totality of the array of grid portions.

11 . The system of claim 10 , wherein the subset of the grid portions comprises:
a central grid portion that is surrounded by outer ones of the array of grid portions; a vertical column of a plurality of the grid portions; or the horizontal row of a plurality of the grid portions.

12 . The system of claim 11 , wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:
selects a one of the central grid portion, the vertical column of grid portions and the horizontal row of grid portions that is most likely to include at least one of the attribute detectors of a multi-attribute query subset plurality of the annotated attributes as the subset portions of the image fields from which the feature vectors are extracted.

13 . The system of claim 12 , wherein the learning the pair-wise correlations is a max-margin training.

14 . The system of claim 13 , wherein the program instructions are provided as a service in a cloud en

Assignees

• IBM

Inventors

• Datta Ankur
• Feris Rogerio S
• Pankanti Sharathchandra U
• Siddiquie Behjat

Classifications

• G06V30/19167

  Active pattern learning · CPC title

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F16/51

  Indexing; Data structures therefor; Storage structures · CPC title

• G06N20/10

  using kernel methods, e.g. support vector machines [SVM] · CPC title