Reranking using confident image samples

Patent metadata

Field	Value
Publication number	US-9384241-B2
Application number	US-201113393509-A
Country	US
Kind code	B2
Filing date	Nov 24, 2011
Priority date	Nov 24, 2011
Publication date	Jul 5, 2016
Grant date	Jul 5, 2016

Abstract

The techniques described herein determine an initial set of ranked images associated with an image-based search query. Based on visual content similarities between images in the initial set of ranked images, the techniques select confident image samples from the initial set of ranked images. The techniques then use the confident image samples to rerank the initial set of ranked images. Accordingly, a search engine uses the confident image samples to promote images that are likely to be relevant to the search query, while demoting images that are not likely to be relevant to the search query. Therefore, the search engine can provide improved relevance-based search results to an image-based search query.

First claim

The invention claimed is:

1. A method comprising:
receiving a search query;
ranking, based at least in part on the search query, a set of images to provide a set of ranked images;
extracting visual features for individual images of the set of ranked images;
determining, by one or more processors and based at least in part on the extracted visual features, a first degree of visual content similarity for individual images of the set of ranked images with respect to one or more other images of the set of ranked images;
implementing, by the one or more processors, a selection algorithm to automatically select at least two confident image samples from the set of ranked images based at least in part on the first degrees of visual content similarity and based at least in part on ranking positions of individual images within the set of ranked images, wherein the at least two confident image samples automatically selected are not consecutively ordered within the set of ranked images;
determining, by the one or more processors and based at least in part on the extracted visual features, a second degree of visual content similarity for individual images of the set of ranked images with respect to the at least two confident image samples; and
reranking the set of ranked images based at least in part on the second degrees of visual content similarity.

2. The method as recited in claim 1 , further comprising providing at least a portion of the set of reranked images as search results in response to receiving the search query.

3. The method as recited in claim 1 , wherein reranking the set of ranked images comprises promoting images that are visually similar to the at least two confident image samples and demoting images that are not visually similar to the at least two confident image samples.

4. The method as recited in claim 1 , wherein the first degree of visual content similarity for a given image is determined based on comparing the extracted visual features associated with the given image to the extracted visual features associated with the one or more other images of the set of ranked images.

5. The method as recited in claim 1 , further comprising implementing a non-negative least squares relaxation algorithm to automatically select the at least two confident image samples from the set of ranked images.

6. The method as recited in claim 1 , further comprising implementing a bounded-variable least squares relaxation algorithm to automatically select the at least two confident image samples from the set of ranked images.

7. The method as recited in claim 1 , further comprising implementing an adapted kernel-based reranking process to rerank the set of ranked images based at least in part on the second degrees of visual content similarity.

8. The method as recited in claim 1 , wherein the set of ranked images associated with the search query is determined based at least in part on matching query terms with textual information corresponding to individual images of the set of ranked images.

9. A system comprising:
one or more processors;
a memory, coupled to the one or more processors, storing:
an image ranking module, operable by the one or more processors, that ranks images associated with a search query based at least in part on textual matching, thereby providing a set of ranked images;
an image feature extraction module, operable by the one or more processors, that extracts visual features from individual ones of the set of ranked images;
a confident sample selection module, operable by the one or more processors, that determines, based at least in part on the extracted visual features, a first degree of visual content similarity for individual ones of the set of ranked images with respect to one or more other images of the set of ranked images and that implements a selection algorithm to automatically select at least two non-consecutively ordered confident image samples based at least in part on the first degrees of visual content similarity and based at least in part on ranking positions of individual images within the set of ranked images; and
a reranking module, operable by the one or more processors, that determines, based at least in part on the extracted visual features, a second degree of visual content similarity for individual ones of the set of ranked images with respect to the at least two non-consecutively ordered confident image samples and that reranks the set of ranked images based at least in part on the second degrees of visual content similarity.

10. The system as recited in claim 9 , wherein the first degree of visual content similarity for a given image is determined based on comparing the extracted visual features associated with the given image to the extracted visual features associated with the one or more other images of the set of ranked images.

11. The system as recited in claim 9 , wherein the confident sample selection module implements a non-negative least squares relaxation algorithm to automatically select the at least two non-consecutively ordered confident image samples from the set of ranked images.

12. The system as recited in claim 9 , wherein the confident sample selection module implements a bounded-variable least squares relaxation algorithm to automatically select the at least two non-consecutively ordered confident image samples from the set of ranked images.

13. The system as recited in claim 9 , wherein the reranking module implements an adapted kernel-based reranking process to rerank the set of ranked images based at least in part on the second degrees of visual content similarity.

14. The system as recited in claim 9 , wherein the at least two non-consecutively ordered confident image samples automatically selected are not consecutively ordered highest ranked images within the set of ranked images.

15. One or more computer storage media comprising instructions that, when executed, configure a computer device to perform operations comprising:
receiving a search query;
ranking a set of images based at least in part on terms used in the search query;
extracting visual features for individual images of the set of ranked images;
determining, based at least in part on the extracted visual features, a first degree of visual content similarity for individual images of the set of ranked images with respect to one or more other images of the set of ranked images;
implementing a selection algorithm to automatically select at least two non-consecutively ordered confident image samples from the set of ranked images based at least in part on the first degrees of visual content similarity and based at least in part on ranking positions of individual images within the set of ranked images;
determining, based at least in part on the extracted visual features, a second degree of visual content similarity for individual images of the set of ranked images with respect to the at least two non-consecutively ordered confident image samples;
reranking the set of ranked images based at least in part on the second degrees of visual content similarity; and
providing at least a portion of the reranked set of images as search results.

16. The one or more computer storage media recited in claim 15 , wherein reranking the set of ranked images comprises promoting images that are visually similar to the at least two non-consecutively ordered confident image samples and demoting images that are not visually similar to the at least two non-consecutively ordered confident image samples.

17. The one or more computer storage media recited in claim 15 , furth

Assignees

• Wang Jingdong
• Li Shipeng
• Morioka Nobuyuki
• Microsoft Technology Licensing Llc

Inventors

• Wang Jingdong
• Li Shipeng
• Morioka Nobuyuki

Classifications

• G06F17/3053 Primary

  Physics · mapped topic

• G06F17/30247

  Physics · mapped topic

• G06F17/30277

  Physics · mapped topic

• G06F16/5838 Primary

  using colour · CPC title

• G06F16/24578 Primary

  using ranking · CPC title