Ranking content items based on preference scores

Patent metadata

Field	Value
Publication number	US-10007732-B2
Application number	US-201514715852-A
Country	US
Kind code	B2
Filing date	May 19, 2015
Priority date	May 19, 2015
Publication date	Jun 26, 2018
Grant date	Jun 26, 2018

Abstract

A set of content items, such as web pages, are identified in response to a query generated by a user. The Identified content items are initially ranked using a ranking scheme. User-interaction data that describes preferences that the user may have towards some of the ranked content items is received. In order to personalize the ranking of the content items for the user, the user-interaction data is used to re-rank the ranked content items in a way that favors content items that are preferred by the user, while also preserving the initial broadly applicable ranking with respect to content items that are not preferred or that are equally preferred by the user.

First claim

What is claimed:

1. A method for re-ordering a plurality of content items using ranking preference data and user-interaction data comprising:
receiving identifiers of a plurality of content items by a computing device, wherein each content item has a position in a first order of the plurality of content items;
determining content items from the plurality of content items that are relevant content items by the computing device;
receiving ranking preference data comprising a first parameter and a second parameter by the computing device, wherein the first parameter indicates a preference of a user for content items that are relevant content items and wherein the second parameter indicates a preference of the user to maintain the first order such that for any two content items that have been determined to be in the same relevant or irrelevant class based on the user-interaction data, the priority for those content items remains the same in a second order as in the first order, wherein relative weights of the first parameter and the second parameter are selectable by the user, and wherein the ranking preference data further comprises a value that represents a preference to rank the content items based on interaction data associated with a different user;
calculating a gain value for each content item based on a sum of preference scores for that content item with respect to each of the other content items of the plurality of content items; and
generating the second order for the plurality of content items using the ranking preference data and the gain value by the computing device.

2. The method of claim 1 , further comprising receiving user-interaction data associated with the user and one or more of the plurality of content items by the computing device.

3. The method of claim 2 , wherein determining content items from the plurality of content items that are relevant content items comprising determining content items from the plurality of content items that are identified by the user-interaction data.

4. The method of claim 2 , wherein the user-interaction data is received from a search engine.

5. The method of claim 2 , wherein the user-interaction data comprises one or more of dwell-time and click data.

6. The method of claim 1 , wherein the second order is different than the first order.

7. The method of claim 1 , further comprising training a ranking model using the second order.

8. The method of claim 1 , wherein generating the second order for the plurality of content items using the ranking preference data comprises:
for each content item:
calculating a preference score for the content item with respect to each of the other content items in the first order based on the first parameter, the second parameter, and whether or not the content item was determined to be a relevant content item; and
calculating the gain value for the content item based on a sum of the calculated preference scores for the content item; and

generating the second order using the gain value calculated for each content item.

9. A method for re-ordering a plurality of content items using ranking preference data and user-interaction data comprising:
receiving identifiers of a plurality of content items by a computing device, wherein each content item has a position in a first order of the plurality of content items;
receiving user-interaction data associated with one or more users and one or more of the plurality of content items by the computing device;
determining content items from the plurality of content items that are relevant content items based on the user-interaction data by the computing device;
determining content items from the plurality of content items are irrelevant content items based on the user-interaction data by the computing device;
receiving ranking preference data associated with a user by the computing device, wherein the ranking preference data comprises a first parameter and a second parameter, wherein the first parameter indicates a preference of the one or more users for content items that are relevant content items and wherein the second parameter indicates a preference of the user to maintain the first order such that for any two content items that have been determined to be in the same relevant or irrelevant class based on the user-interaction data, the priority for those content items remains the same in a second order as in the first order, wherein relative weights of the first parameter and the second parameter are selectable by the user, and wherein the ranking preference data further comprises a value that represents a preference to rank the content items based on interaction data associated with a different user;
calculating a gain value for each content item based on a sum of preference scores for that content item with respect to each of the other content items of the plurality of content items; and
generating the second order for the plurality of content items by the computing device based on the determined content items that are relevant content items, the determined content items that are irrelevant content items, the ranking preference data, and the gain value.

10. The method of claim 9 , wherein the content items comprise one or more of web pages, images, videos, or documents.

11. The method of claim 9 , further comprising training a ranking model using the second order.

12. The method of claim 9 , wherein the second order is different than the first order.

13. The method of claim 9 , wherein the user-interaction data comprises one or more of dwell-time and click data.

14. The method of claim 9 , wherein the ranking preference data is specified by the one or more users.

15. The method of claim 9 , wherein generating the second order for the plurality of content items based on the determined content items that are relevant content items, the determined content items that are irrelevant content items, and the ranking preference data comprises:
for each content item:
calculating a preference score for the content item with respect to each of the other content items based on the first parameter, the second parameter; and whether the content item was determined to be a relevant content item or an irrelevant content item;
calculating the gain value for the content item based on a sum of the calculated preference scores for the content item; and
generating the second order using the gain calculated for each content item.

16. A system for re-ordering a plurality of web pages using ranking preference data and user-interaction data comprising:
at least one computing device configured to generate a query for a user; and
a user-interaction ranker configured to:
receive identifiers of a plurality of web pages, wherein each web page has a position in a first order of the plurality of web pages and each web page is responsive to the generated query;
determine web pages from the plurality of web pages that are relevant web pages;
receive ranking preference data comprising a first parameter and a second parameter, wherein the first parameter indicates a preference of the user for web pages that are relevant web pages and wherein the second parameter indicates a preference of the user to maintain the first order such that for any two web pages that have been determined to be in the same relevant or irrelevant class based on the user-interaction data, the priority for those web pages remains the same in a second order as in the first order, wherein relative weights of the first parameter and the second parameter are selectable by the user, and wherein the ranking preference dat

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bennett Paul
• Shokouhi Milad
• Caruana Richard A

Classifications

• G06F16/24578

  using ranking · CPC title

• G06F16/9535 Primary

  Search customisation based on user profiles and personalisation · CPC title

• G06F17/30867 Primary

  Physics · mapped topic

• G06F17/3053

  Physics · mapped topic