Method and system for predicting search results quality in vertical ranking

Patent metadata

Field	Value
Publication number	US-2016019213-A1
Application number	US-201414332501-A
Country	US
Kind code	A1
Filing date	Jul 16, 2014
Priority date	Jul 16, 2014
Publication date	Jan 21, 2016
Grant date	—

Abstract

Methods, systems and programming for predicting search results quality. In one example, a search query is received from a user. A plurality of search results are obtained from a content source based on the search query. The plurality of search results are ranked based on their relevance scores with respect to the search query. A distribution of the relevance scores of the plurality of search results is normalized in each position of the ranking. A metric of the content source is computed based on the normalized distribution of the relevance scores. The metric indicates a relevance between the plurality of search results and the search query.

First claim

We claim:

1 . A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for predicting search results quality, the method comprising the steps of:
receiving a search query from a user: obtaining a plurality of search results from a content source based on the search query, wherein the plurality of search results are ranked based on their relevance scores with respect to the search query; normalizing a distribution of the relevance scores of the plurality of search results in each position of the ranking; and computing a metric of the content source based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the plurality of search results and the search query.

2 . The method of claim 1 , wherein normalizing a distribution of the relevance scores includes:
in each position of the ranking, computing a normalized relevance, score of the respective search result based on a mean and a standard deviation of relevance scores in the position.

3 . The method of claim 2 , wherein the mean and the standard deviation of relevance scores in the position are obtained by:
obtaining a plurality of sample query results from the content source based on each of a plurality of sample queries, each of the sample query results being ranked in the position; and computing the mean and the standard deviation of the plurality of sample queries results in the position.

4 . The method of claim 1 , wherein normalizing a distribution of the relevance scores comprises:
building an order-statistic model based on a first set of the search results; and generating estimated relevance scores of a second set of the search results based on the order-statistic model.

5 . The method of claim 4 , wherein computing a metric includes:
comparing the relevance scores of the second set of the search results with the estimated relevance scores of the second set of the search results.

6 . The method of claim 4 , wherein the first set of the search results are ranked lower than the second set of the search results.

7 . The method of claim 4 , wherein the order-statistic model is built based on the relevance scores of the first set of the search results and the positions of the first set of the search results in the ranking.

8 . The method of claim 4 , wherein the estimated relevance scores of the second set of the search results are generated based on the positions of the second set of the search results in the ranking.

9 . The method of claim 4 , wherein the relevance scores of the first set of the search results are approximated by a normal distribution.

10 . The method of claim 1 , wherein the content source includes a vertical in vertical search.

10 . A system having at least one processor storage, and a communication platform for predicting search results quality, the system comprising:
a search engine configured to receive a search query from a user and obtain a plurality of search results from a content source based on the search query, wherein the plurality of search results are ranked based on their relevance scores with respect to the search query; a normalization module configured to normalize a distribution of the relevance scores of the plurality of search results in each position of the ranking; and a ranking module configured to compute a metric of the content source based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the plurality of search results and the search query.

12 . The system of claim 11 , wherein the ranking engine includes a normalization module configured to, in each position of the ranking, compute a normalized relevance score of the respective search result based on a mean and a standard deviation of relevance scores in the position.

3 . The system of claim 12 , wherein the mean and the standard deviation of relevance scores in the position are obtained by:
obtaining a plurality of sample query results from the content source based on each of a plurality of sample queries, each of the sample query results being ranked in the position; and computing the mean and the standard deviation of the plurality of sample queries results in the position.

14 . The system of claim 11 , wherein the ranking engine includes a normalization module configured to
build an order-statistic model based on a first set of the search results; and generate estimated relevance scores of a second set of the search results based on the order-statistic model.

15 . The system of claim 14 , wherein the normalization module is further configured to compare the relevance scores of the second set of the search results with the estimated relevance scores of the second set of the search results.

16 . The system of claim 14 , wherein the first set of the search results are ranked lower than the second set of the search results.

17 . The system of claim 14 , wherein the order-statistic model is built based on the relevance scores of the first set of the search results and the positions of the first set of the search results in the ranking.

18 . The system of claim 14 , wherein the estimated relevance scores of the second set of the search results are generated based on the positions of the second set of the search results in the ranking.

19 . The system of claim 14 , wherein the relevance scores of the first set of the search results are approximated by a normal distribution.

20 . A non-transitory machine-readable medium having information recorded thereon for predicting, search results quality, wherein the information, when read by the machine, causes the machine to perform the following:
receiving a search query from a user; obtaining a plurality of search results from a content source based on the search query, wherein the plurality of search results are ranked based on their relevance scores with respect to the search query; normalizing a distribution of the relevance scores of the plurality of search results in each position of the ranking; and computing a metric of the content source based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the plurality of search results and the search query.

Assignees

• Yahoo Inc

Inventors

• Carmel David
• Wolff Ran

Classifications

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/9538 Primary

  Presentation of query results · CPC title

• G06F17/30864

  Physics · mapped topic

• G06F17/3053 Primary

  Physics · mapped topic