Search service advertisement selection

Patent metadata

Field	Value
Publication number	US-9589277-B2
Application number	US-201314145422-A
Country	US
Kind code	B2
Filing date	Dec 31, 2013
Priority date	Dec 31, 2013
Publication date	Mar 7, 2017
Grant date	Mar 7, 2017

Abstract

Methods, computer systems, and computer storage media are provided for evaluating information retrieval (IR) such as search query results (including advertisements) by a machine learning scorer. In an embodiment, a set of features is derived from a query and a machine learning algorithm is applied to construct a linear model of (query, ads) for scoring by maximizing a relevance metric. In an embodiment, the machine learned scorer is adapted for use with WAND algorithm based ad selection.

First claim

The invention claimed is:

1. One or more computer-storage hardware devices having computer-executable instructions embodied thereon that, when executed by a computing device, cause the computing device to perform a method of determining relevant advertisements with a machine learning model based on a query, the method comprising:
receiving training data for training a machine learning model, the training data comprising information indicating relevance between a set of reference and target documents, the documents comprising one or more training terms;
determining one or more training data feature sets and corresponding training data feature values from the training terms of the training data, each feature set and values associated with a training term;
generating, based on the training data feature sets and corresponding values, a cross-joined feature set of features comprising a combined first feature of a reference-document data feature and a second, different feature of a target-document feature;
training the machine learning model by determining a set of weight parameters for each of the training terms by applying the one or more of the training feature sets, the cross-joined feature set, and values associated with the training term to the model;
in response to a query, utilizing the model to determine a final measure of relevance between a query-target document and the query; and
based on the final measure of relevance, causing the query-target document to be presented.

2. The one or more computer-storage hardware devices of claim 1 , further comprising:
receiving advertisement information comprising at least one advertisement document, each document including one or more advertising terms;
based on the set of weight parameters, determining a first upper bound parameter value for each of the one or more advertising terms; and
storing the first upper bound parameter value for each of the one or more advertising terms.

3. The one or more computer-storage hardware devices of claim 2 , further comprising:
the query comprising one or more query-terms;
determining a set of query-features and query-bins from the query;
from the query and advertising information, identifying ad-query pairs to be evaluated for relevance to each other, thereby forming a set of ad-query pairs;
for each ad-query pair, based on the determined set of query-features and first upper bound parameter value for each advertising term:
(1) determining a second upper bound parameter value for the ad-query pair;
(2) determining a rough relevance for the ad-query pair; and
(3) determining a difference between the rough relevance measurement and a threshold value to classify the ad-query pair as not relevant or potentially relevant.

4. The one or more computer-storage hardware devices of claim 3 , further comprising:
for each potentially relevant ad-query pair, determining a second measure of relevance by applying the machine learning model to the potentially relevant ad-query pair.

5. The one or more computer-storage hardware devices of claim 4 , further comprising selecting one or more advertisements for presentation based on the second measure of relevance.

6. The one or more computer-storage hardware devices of claim 4 , further comprising ranking the ad-query pairs based on the second measure of relevance for each ad-query pair.

7. The one or more computer-storage hardware devices of claim 1 , wherein the training data comprises one of user click data, publisher click data and historical ad-query usage information.

8. The one or more computer-storage hardware devices of claim 3 , wherein the threshold value comprises a top N most relevant ads heap threshold based at least on the term frequency or inverse document frequency of a term.

9. The one or more computer-storage hardware devices of claim 1 , wherein the machine learning model type is non-linear.

10. The one or more computer-storage hardware devices of claim 1 , wherein the machine learning model comprises an L-1 regularized logistic regression model.

11. The one or more computer-storage hardware devices of claim 10 , wherein the logistic regression model is of the form:

min
w

⁢

{

-

∑
i

⁢

{

log
⁡

[

P
(
R


⁢

(

q
,
a

)

i

;
w

]

*

y
i

+

log
⁡

[

1
-

P
(
R


⁢

(

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Zhang Bruce
• Mao Jianchang
• Shen Yuan

Classifications

• G06Q30/0256 Primary

  User search · CPC title

• G06N99/005

  Physics · mapped topic

• G06N20/00

  Machine learning · CPC title