Discriminative embedding of local color names for object retrieval and classification

Patent metadata

Field	Value
Publication number	US-9600738-B2
Application number	US-201514680490-A
Country	US
Kind code	B2
Filing date	Apr 7, 2015
Priority date	Apr 7, 2015
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

A system and method enable similarity measures to be computed between pairs of images and between a color name and an image in a common feature space. Reference image representations are generated by embedding color name descriptors for each reference image in the common feature space. Color name representations for different color names are generated by embedding synthesized color name descriptors in the common feature space. For a query including a color name, a similarity is computed between its color name representation and one or more of the reference image representations. For a query which includes a query image, a similarity is computed between a representation of the query image and one or more of reference image representations. The method also enables combined queries which include both a query image and a color name to be performed. One or more retrieved reference images, or information based thereon, is then output.

First claim

What is claimed is:

1. A retrieval method comprising:
a) for each of at least one reference image, providing a reference image representation which has been generated by embedding color name descriptors for patches of the reference image into a feature space, each color name descriptor including an element for each of the color names in the set of color names;
b) for each of at least one color name in a set of color names, providing a color name representation which has been generated by embedding a synthesized color name descriptor in the same feature space as the reference images, each synthesized color name descriptor including an element for each of the color names in the set of color names, each embedded synthesized color name descriptor having a same number of dimensions as each of the reference image representations;
c) receiving a query comprising at least one of:
a color name selected from the set of color names, and
a query image;

d) for a query which includes a color name, retrieving the respective color name representation;
e) for a query which includes a query image, providing a query image representation which has been generated by embedding color name descriptors for the query image in the multidimensional feature space;
f) computing a similarity between:
at least one of the at least one reference image representations, and
at least one of the color name representation and the query image representation; and

g) outputting information based on the computed similarity,
wherein at least one of a), b), c), d), e), f), and g) is performed with a processor.

2. The method of claim 1 , wherein the provided color name representation has been generated by:
synthesizing a local color name descriptor;
generating a color name representation comprising embedding the synthesized local color name descriptor with a first embedding function.

3. The method of claim 2 , wherein the synthesized local color name descriptor includes a non-zero value for only one of a plurality of elements, each element corresponding to a color name in the set of color names.

4. The method of claim 2 , wherein the first embedding function is a Fisher Vector embedding function.

5. The method of claim 1 , wherein the provided reference image representation has been generated by:
assigning to each of a set of pixels of the reference image, a probability distribution over a predefined set of color names;
for each of a plurality of patches of the reference image:
generating a local color name descriptor, comprising aggregating the pixel probability distributions, and
generating an embedded patch descriptor comprising embedding the local color name descriptor with a second embedding function; and

generating the reference image representation, comprising aggregating the embedded patch descriptors.

6. The method of claim 5 , wherein the second embedding function is a Fisher Vector embedding function.

7. The method of claim 5 , wherein the reference image representation has been generated by:
defining a plurality of patches of the reference image, each patch including a plurality of cells;
for pixels of each cell aggregating the pixel probability distributions; and
for each of the plurality of patches of the reference image, generating the local color name descriptor, comprising concatenating the aggregated pixel probability distributions of the cells in the patch.

8. The method of claim 1 , wherein the query includes a query image and the generating of the query image representation comprises:
assigning to each of a set of pixels of the query image, a probability distribution over the set of color names;
for each of a plurality of patches of the query image:
generating a local color name descriptor, comprising aggregating the pixel probability distributions, and
generating an embedded patch descriptor comprising embedding the local color name descriptor with a second embedding function; and

generating a query image representation, comprising aggregating the embedded patch descriptors.

9. The method of claim 8 , wherein the query image representation is generated by:
defining a plurality of patches of the query image, each patch including a plurality of cells;
for each cell, aggregating the pixel probability distributions of the pixels of the cell; and
for each of the plurality of patches of the query image, generating the local color name descriptor, comprising concatenating the aggregated pixel probability distributions of the cells in the respective patch.

10. The method of claim 9 , wherein the aggregating of the pixel probability distributions is performed by max pooling aggregation.

11. The method of claim 1 , wherein the reference image comprises a plurality of reference images.

12. The method of claim 1 , wherein the at least one color name includes at least four color names.

13. The method of claim 1 , wherein the at least one color name includes no more than twenty color names.

14. The method of claim 1 , wherein the received query includes a color name and the similarity is computed based on the color name representation.

15. The method of claim 14 , wherein the received query further includes a query image, and the similarity is computed based on the query image representation and the color name representation.

16. The method of claim 1 , wherein the computing of the similarity comprises computing a cosine similarity between the at least one of the at least one reference image representation, and the at least one of the color name representation and the query image representation.

17. A computer program product comprising non-transitory memory storing instructions which when executed by a computer, perform the method of claim 1 .

18. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

19. A retrieval method comprising:
a) for each of at least one reference image, providing a reference image representation which has been generated by embedding color name descriptors for the reference image in a feature space;
b) for each of at least one color name in a set of color names, providing a color name representation which has been generated by embedding a synthesized color name descriptor in the same feature space as the embedded color name descriptors of the reference image;
c) providing for receiving a query comprising at least one of:
a color name selected from the set of color names, and
a query image;

d) for a query which includes a color name, retrieving the respective color name representation;
e) for a query which includes a query image, providing a query image representation which has been generated by embedding color name descriptors for the query image in the feature space;
f) computing a similarity between:
at least one of the at least one reference image representations, and
at least one of the color name representation and the query image representation,
wherein when the received query includes both a color name selected from the set of color names and a query image, the similarity is computed based on the query image representation and the color name representation; and

g) outputting information based on the computed similarity,
wherein at least one of a), b), c), d), e), f), and g) is performed with a processor.

20. A system comprising:
an image representation generation component which generates representations of images by embedding color nam

Assignees

• Xerox Corp

Inventors

• Murray Naila
• Perronnin Florent C

Classifications

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V10/56

  relating to colour · CPC title

• G06F16/50 Primary

  of still image data · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title