Using feedback to re-weight candidate features in a streaming environment

Patent metadata

Field	Value
Publication number	US-2018232702-A1
Application number	US-201715851584-A
Country	US
Kind code	A1
Filing date	Dec 21, 2017
Priority date	Feb 16, 2017
Publication date	Aug 16, 2018
Grant date	—

Abstract

Techniques for dynamically altering weights to re-weight candidate features of a candidate search and ranking model in a streaming environment are described. In an embodiment, a disclosed system obtains desired hire documents using a search query specifying parameters. Additionally, the system extracts desired hire-based features from the documents, with the features corresponding to the parameters. Moreover, the system inputs the features to a combined ranking model that is trained by a machine learning algorithm to output a ranking score for each of the documents, with the combined ranking model including weights assigned to each of the features. Furthermore, the system ranks the desired hire documents based on the ranking scores and displays top ranked documents. Then, feedback is received regarding the top ranked documents, and the weights assigned to each of the features are dynamically trained to alter the weights assigned to each of the features based on the feedback.

First claim

What is claimed is:

1 . A computer system, comprising:
one or more processors; and a non-transitory computer readable storage medium storing instructions that when executed by the one or more processors cause the computer system to perform operations comprising:
obtaining one or more desired hire documents using a search query specifying one or more parameters;
extracting one or more desired hire-based features from the one or more desired hire documents, the one or more desired hire-based features corresponding to the one or more parameters;
inputting the one or more desired hire-based features to a combined ranking model, the combined ranking model trained by a machine learning algorithm to output a ranking score for each of the one or more desired hire documents, the combined ranking model including weights assigned to each of the one or more desired hire-based features;
ranking the one or more desired hire documents based on the ranking scores;
causing the one or more top ranked documents to be displayed on a display device;
receiving feedback regarding the displayed one or more top ranked documents; and
dynamically training the weights assigned to each of the one or more desired hire-based features to alter the weights assigned to each of the one or more desired hire-based features based on the received feedback.

2 . The system of claim 1 , wherein the desired hire documents are member profiles in a social networking service.

3 . The system of claim 1 , wherein the instructions further cause the system to:
repeating the ranking based on the altered weights; updating the display of the one or more top ranked documents on the display device, the updating being based on the repeating; receiving additional feedback regarding the updated display of the one or more top ranked documents; and dynamically re-training the weights assigned to each of the one or more desired hire-based features to alter the weights assigned to each of the one or more desired hire-based features based on the additional feedback.

4 . The system of claim 1 , wherein the feedback includes explicit feedback received from a user interacting with the display of the one or more top ranked documents.

5 . The system of claim 4 , wherein the explicit feedback includes one of: acceptance, deferral, or rejection of a candidate corresponding to one of the displayed one or more top ranked documents.

6 . The system of claim 1 , wherein the feedback includes implicit feedback received from a user interacting with the display of the one or more top ranked documents.

7 . The system of claim 6 , wherein the implicit feedback includes measured metrics corresponding to one or more of dwell time on the displayed one or more top ranked documents, profile sections viewed of the displayed one or more top ranked documents, and a number of revisits to the displayed one or more top ranked documents.

8 . A computer-implemented method, comprising:
obtaining one or more desired hire documents using a search query specifying one or more parameters; extracting one or more desired hire-based features from the one or more desired hire documents, the one or more features corresponding to the one or more parameters; inputting the one or more desired hire-based features to a combined ranking model, the combined ranking model trained by a machine learning algorithm to output a ranking score for each of the one or more desired hire documents, the combined ranking model including weights assigned to each of the one or more desired hire-based features; ranking the one or more desired hire documents based on the ranking scores; causing display of one or more top ranked documents on a display device; receiving feedback regarding the displayed one or more top ranked documents; and dynamically training the weights assigned to each of the one or more desired hire-based features to alter the weights assigned to each of the one or more desired hire-based features based on the received feedback.

9 . The method of claim 8 , wherein the desired hire documents are member profiles in a social networking service.

10 . The method of claim 8 , further comprising:
repeating the ranking based on the altered weights; updating the display of the one or more top ranked documents on the display device, the updating being based on the repeating; receiving additional feedback regarding the updated display of the one or more top ranked documents; and dynamically re-training the weights assigned to each of the one or more desired hire-based features to alter the weights assigned to each of the one or more desired hire-based features based on the additional feedback.

11 . The method of claim 8 , wherein the feedback includes explicit feedback received from a user interacting with the display of the one or more top ranked documents.

12 . The method of claim 11 , wherein the explicit feedback includes one of: acceptance, deferral, or rejection of a candidate corresponding to one of the displayed one or more top ranked documents.

13 . The method of claim 8 , wherein the feedback includes implicit feedback received from a user interacting with the display of the one or more top ranked documents.

14 . The method of claim 13 , wherein the implicit feedback includes measured metrics corresponding to one or more of dwell time on the displayed one or more top ranked documents, profile sections viewed of the displayed one or more top ranked documents, and a number of revisits to the displayed one or more top ranked documents.

15 . A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:
obtaining one or more desired hire documents using a search query specifying one or more parameters; extracting one or more desired hire-based features from the one or more desired hire documents, the one or more features corresponding to the one or more parameters; inputting the one or more desired hire-based features to a combined ranking model, the combined ranking model trained by a machine learning algorithm to output a ranking score for each of the one or more desired hire documents, the combined ranking model including weights assigned to each of the one or more desired hire-based features; ranking the one or more desired hire documents based on the ranking scores; causing display of one or more top ranked documents on a display device; receiving feedback regarding the displayed one or more top ranked documents; and dynamically training the weights assigned to each of the one or more desired hire-based features to alter the weights assigned to each of the one or more desired hire-based features based on the received feedback.

16 . The non-transitory machine-readable storage medium of claim 15 , wherein the desired hire documents are member profiles in a social networking service.

17 . The non-transitory machine-readable storage medium of claim 15 , wherein the operations further comprise:
repeating the ranking based on the altered weights; updating the display of the one or more top ranked documents on the display device, the updating being based on the repeating; receiving additional feedback regarding the updated display of the one or more top ranked documents; and dynamically re-training the weights assigned to each of the one or more desired hire-based features to alter the weights assigned to each of the one or more desired hire-based features based on the additional feedback.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Dialani Vijay
• Geyik Sahin Cem
• Wu Xianren
• Gupta Abhishek

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F16/93

  Document management systems · CPC title

• G06Q10/1053 Primary

  Employment or hiring · CPC title

• G06F17/30011

  Physics · mapped topic