Label-embedding view of attribute-based recognition

Patent metadata

Field	Value
Publication number	US-10331976-B2
Application number	US-201313923639-A
Country	US
Kind code	B2
Filing date	Jun 21, 2013
Priority date	Jun 21, 2013
Publication date	Jun 25, 2019
Grant date	Jun 25, 2019

Abstract

In image classification, each class of a set of classes is embedded in an attribute space where each dimension of the attribute space corresponds to a class attribute. The embedding generates a class attribute vector for each class of the set of classes. A set of parameters of a prediction function operating in the attribute space respective to a set of training images annotated with classes of the set of classes is optimized such that the prediction function with the optimized set of parameters optimally predicts the annotated classes for the set of training images. The prediction function with the optimized set of parameters is applied to an input image to generate at least one class label for the input image. The image classification does not include applying a class attribute classifier to the input image.

First claim

The invention claimed is:

1. A non-transitory storage medium storing instructions readable and executable by an electronic data processing device to perform an image cataloging method including the operations of:
representing classes of a set of classes Y={y j , j=1, . . . , C} by class attribute vectors φ(y j ) where φ is an embedding function that embeds a class in an attribute space of dimensionality E where each dimension a i , i=1, . . . , E of the attribute space corresponds to a class attribute of a set of E class attributes;
representing training images x n labeled by respective training image class labels y n as θ(x n ) where θ is an embedding function that embeds an image in an image feature space of dimensionality D image features, the training images not being labeled with class attributes of the attribute space of dimensionality E;
optimizing, respective to the training images, a set of parameters w of a prediction function

f
⁡

(

x
;
w

)

=

arg
⁢

⁢

max

y
∈
Y

⁢

⁢

{

θ
⁡

(
x
)

′

⁢
W
⁢

⁢

φ
⁡

(
y
)

}

where x denotes an image and y denotes a class of the set of classes Y and where W is a D×E matrix representing the set of parameters w of the prediction function ƒ(x;w) wherein the training images respective to which the set of parameters w of the prediction function ƒ(x;w) is optimized do not include training images that are labeled with class attributes of the set of E class attributes;
applying the prediction function ƒ(x;w) with the optimized set of parameters W to an input image to generate at least one class label for the input image; and
cataloging the input image using the at least one class label for the input image, the cataloging including storing the input image or a pointer to the input image in an images database structured by image class.

2. The non-transitory storage medium as set forth in claim 1 wherein the applying does not include applying image attribute classifiers to the input image.

3. The non-transitory storage medium as set forth in claim 1 wherein the class attributes of the attribute space include image attributes.

4. The non-transitory storage medium as set forth in claim 3 wherein the set of classes Y comprises a hierarchy of classes and the class attributes of the attribute space further include class hierarchy attributes derived from the hierarchy of classes.

5. The non-transitory storage medium as set forth in claim 1 wherein the set of classes Y comprises a hierarchy of classes and the class attributes of the attribute space include class hierarchy attributes derived from the hierarchy of classes.

6. The non-transitory storage medium as set forth in claim 1 wherein the applying further includes:
generating class attribute scores for the input image for class attributes of the attribute space based on θ(x in )′W where x in denotes the input image.

7. The non-transitory storage medium as set forth in claim 1 wherein the optimizing includes optimizing at least one of a ranking objective, a multiclass objective, and a regression objective.

8. The non-transitory storage medium of claim 1 further comprising:
receiving assignments of class attributes of the set of class attributes to classes of the set of classes;
wherein the representing of the classes by class attribute vectors φ(y j ) uses the received assignments of class attributes to the classes.

9. An apparatus for performing image retrieval, the apparatus comprising:
an electronic data processing device programmed to perform a method including the operations of:
receiving assignments of class attributes of a set of class attributes to classes of a set of classes;
embedding classes of the set of classes Y={y j , j=1, . . . , C} in an attribute space of dimensionality E using the received assignments of class attributes to the classes, where each dimension a i , i=1, . . . , E of the attribute space corresponds to a class attribute of the set of E class attributes;
embedding training images x n labeled by respective training image class labels y n in an image feature space of dimensionality D image features wherein none of the training images are labeled with class attributes of the attribute space;
optimizing a set of parameters w of a prediction function

y
=

f
⁡

(

x
;
w

)

=

arg
⁢

⁢
max

y
∈
Y

⁢

{

θ
⁡

(
x
)

′

⁢
W
⁢

⁢

φ
⁡

(
y
)

}

 operating in the image feature space an

Assignees

• Xerox Corp

Inventors

• Akata Zeynep
• Perronnin Florent C
• Harchaoui Zaid
• Schmid Cordelia L

Classifications

• G06V10/7715 Primary

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06F2218/00

  Aspects of pattern recognition specially adapted for signal processing · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06F18/2431

  Multiple classes · CPC title

• G06F18/2132

  based on discrimination criteria, e.g. discriminant analysis · CPC title