Position debiasing using inverse propensity weight in machine-learned model

Patent metadata

Field	Value
Publication number	US-2020401643-A1
Application number	US-201916449135-A
Country	US
Kind code	A1
Filing date	Jun 21, 2019
Priority date	Jun 21, 2019
Publication date	Dec 24, 2020
Grant date	—

Abstract

In an example embodiment, position bias is addressed by introducing an inverse propensity weight into a loss function used to train a machine-learned model. This inverse propensity weight essentially increases the weight of candidates in the training data that were presented lower in a list of candidates. This achieves the benefit of counteracting the position bias and increases the effectiveness of the machine-learned model in generating scores for future candidates. In a further example embodiment, a function is generated for the inverse propensity weight based on responses to contact requests from recruiters. In other words, while the machine learned-model may factor in both the likelihood that a recruiter will want to contact a candidate and the likelihood that a candidate will respond to such a contact, the function generated for the inverse propensity weight will be based only on training data where the candidate actually responded to a contact.

First claim

What is claimed is:

1 . A system for returning search results in an online computer system, the system comprising:
a computer readable medium having instructions stored there on, which, when executed by a processor, cause the system to: generate a first inverse propensity weight function from a total number of responses generated by members of an online service in response to communication requests from other members, at each position in a ranking of returned member profiles; for each sample member profile in a plurality of sample member profiles:
parse the sample member profile and the activity and usage information pertaining to actions taken by a member corresponding to the sample member profile in an online service to extract a first set of one or more features;
parse an associated query to obtain one or more query features;
feed sample member labels, the extracted first set of one or more features and the extracted one or more query features into a first machine learning algorithm to train a machine-learned model to output a score indicative of a probability that a searcher will select a potential search result corresponding to the sample member profile and a probability that a member corresponding to the sample member profile will respond to a communication from a searcher, the training including using the first inverse propensity weight function to evaluate a loss function.

2 . The system of claim 1 , wherein the instructions further cause the system to:
obtain a plurality of candidate search results in response to a query corresponding to a searcher; for each candidate search result from the plurality of candidate search results:
obtain activity and usage information for a member corresponding to the candidate search result;
parse the candidate search result and the activity and usage information for the member corresponding to the candidate search result to extract a second set of one or more features;
parse the query to obtain one or more query features;
feed the extracted second set of one or more features and one or more query features into the machine-learned model, outputting a score for the candidate search result;
rank the plurality of candidate search results by their corresponding scores; and return one or more of the plurality of candidate search results based on the ranking.

3 . The system of claim 1 , wherein the first inverse propensity weight function is generated only for positions in the ranking higher than or equal to a preset position; and
wherein instructions further comprise: generating a second inverse propensity weight function, for positions in the ranking lower than the preset position, from a metric other than the total number of responses generated by members in response to communication requests from other members, at each position in a ranking of returned member profiles; and wherein the training further uses the second inverse propensity weight function to evaluate the loss function.

4 . The system of claim 3 , wherein the second inverse propensity weight function is generated by modeling a curve to fit a curve generated by the first inverse propensity weight function.

5 . The system of claim 3 , wherein the second inverse propensity weight function assigns a weight; to all positions in the ranking lower than the preset position, that is equal to a weight assigned by the first inverse propensity weight function to the position the ranking equal to the preset position.

6 . The system of claim 1 , wherein the machine-learned model is a deep neural network.

7 . The system of claim 1 , wherein the machine-learned model is a convolutional neural network.

8 . A computer-implemented method for returning search results in an online computer system, the method comprising:
generating a first inverse propensity weight function from a total number of responses generated by members of an online service in response to communication requests from other members, at each position in a ranking of returned member profiles; for each sample member profile in a plurality of sample member profiles:
parsing the sample member profile and the activity and usage information pertaining to actions taken by a member corresponding to the sample member profile in an online service to extract a first set of one or more features;
parsing an associated query to obtain one or more query features;
feeding sample member labels, the extracted first set of one or more features and the extracted one or more query features into a first machine learning algorithm to train a machine-learned model to output a score indicative of a probability that a searcher will select a potential search result corresponding to the sample member profile and a probability that a member corresponding to the sample member profile will respond to a communication from a searcher, the training including using the first inverse propensity weight function to evaluate a loss function.

9 . The method of claim 8 , wherein the instructions further cause the system to:
obtain a plurality of candidate search results in response to a query corresponding to a searcher; for each candidate search result from the plurality of candidate search results:
obtain activity and usage information for a member corresponding to the candidate search result;
parse the candidate search result and the activity and usage information for the member corresponding to the candidate search result to extract a second set of one or more features;
parse the query to obtain one or more query features;
feed the extracted second set of one or more features and one or more query features into the machine-learned model, outputting a score for the candidate search result;
rank the plurality of candidate search results by their corresponding scores; and return one or more of the plurality of candidate search results based on the ranking.

10 . The method of claim 8 , wherein the first inverse propensity weight function is generated only for positions in the ranking higher than or equal to a preset position; and
wherein instructions further comprise: generating a second inverse propensity weight function, for positions in the ranking lower than the preset position, from a metric other than the total number of responses generated by members in response to communication requests from other members, at each position in a ranking of returned member profiles; and wherein the training further uses the second inverse propensity weight function to evaluate the loss function.

11 . The method of claim 10 , wherein the second inverse propensity weight function is generated by modeling a curve to fit a curve generated by the first inverse propensity weight function.

12 . The method of claim 10 , wherein the second inverse propensity weight function assigns a weight, to all positions in the ranking lower than the preset position, that is equal to a weight assigned by the first inverse propensity weight function to the position the ranking equal to the preset position.

13 . The method of claim 8 ; wherein the machine-learned model is a deep neural network.

14 . The method of claim 8 , wherein the machine-learned model is a convolutional neural network.

15 . A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:
generating a first inverse propensity weight function from a total number of responses generated by members of an online service in response to communication requests from other members, at ea

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Liu Dan
• Hewlett Daniel Sairom Krishnan
• Guo Qi

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title