Dynamic slotting using blending model

Patent metadata

Field	Value
Publication number	US-11763264-B2
Application number	US-202117486577-A
Country	US
Kind code	B2
Filing date	Sep 27, 2021
Priority date	Sep 27, 2021
Publication date	Sep 19, 2023
Grant date	Sep 19, 2023

Abstract

Sponsored and organic pieces of content are displayed in accordance with a blending model that is used to first identify a pattern of slots to which to assign either sponsored or organic pieces of content. This blending model is applied to a combination of both sponsored and non-sponsored pieces of content being considered for display to a user. This pattern only determines the slot assignments. The actual ranking of the pieces of content, and more particularly the actual ranking of the organic pieces of content, is determined by an ordering other than the ranking determined by the blending model, such as by using the original ordering of the second list. The pieces of content are then displayed in the order of this actual ranking, but in the slots indicated as having been assigned to be either sponsored or organic in the pattern determined by the blending model.

First claim

What is claimed is:

1. A system comprising:
a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the system to perform operations comprising:
accessing a first list of sponsored pieces of content, the first list ordered in a first order;
accessing a second list of organic pieces of content, the second list ordered in a second order;
merging the first list and the second list into a merged list including a plurality of pieces of content;
calculating blended engagement scores for the plurality of pieces of content by:
passing the plurality of pieces of content to an engagement model, the engagement model being a machine-learned model trained by a machine learning algorithm to output an engagement score indicative of a likelihood that a user will engage with a corresponding piece of content if presented with the corresponding piece of content in a user interface; and
combining the engagement scores for the plurality of pieces of content with bids to arrive at blended engagement scores for the plurality of pieces of content;
ranking the plurality of pieces of content using a first ranking algorithm, based on their respective blended engagement scores, resulting in the plurality of pieces of content having organic pieces of content ranked in a third order different than the second order;
creating a pattern for display of the pieces of content by indicating, at each slot in the pattern, whether the corresponding piece of content in the ranking of the plurality of pieces of content is sponsored or organic;
assigning one or more sponsored pieces of content, from the first list to slots in the pattern, that have been identified as sponsored;
assigning one or more organic pieces of content, from the second list to slots in the pattern, that have been identified as organic, the assigning the organic pieces of content in a fourth order different than the third order, the fourth ordering created using a second ranking algorithm different than the first ranking algorithm; and
causing display, in a graphical user interface, of the assigned one or more sponsored pieces of content and the assigned one or more organic pieces of content in the slots in the pattern in which they have been assigned.

2. The system of claim 1 , wherein for sponsored pieces of content the bid is an amount a content poster has agreed to pay for an impression of the corresponding sponsored piece of content.

3. The system of claim 1 , wherein for organic pieces of content the bid is a shadow bid indicative of a hypothetical value of an impression of the corresponding organic piece of content.

4. The system of claim 1 , wherein the merging comprises removing any pieces of content in the second list that are duplicative with a piece of content in the first list.

5. The system of claim 1 , wherein the fourth order is the same as the second order.

6. The system of claim 1 , wherein the ranking comprises ranking in accordance with one or more constraints on placement locations for sponsored and/or organic pieces of content.

7. The system of claim 6 , wherein the one or more constraints include a maximum number of consecutive sponsored pieces of content beginning at a highest slot of a display area.

8. The system of claim 6 , wherein the one or more constraints include a maximum number of consecutive sponsored pieces of content.

9. The system of claim 6 , wherein the one or more constraints include a minimum gap between two blocks of a set block size number of sponsored pieces of content.

10. The system of claim 6 , wherein the one or more constraints include a total maximum number of sponsored pieces of content in a displayed page.

11. The system of claim 6 , wherein the one or more constraints include a minimum number of consecutive organic pieces of content beginning at a highest slot of a display area.

12. The system of claim 6 , wherein the one or more constraints include a maximum number of sponsored pieces of content from any one job poster or company per page.

13. The system of claim 1 , wherein the pieces of content are job listings.

14. The system of claim 1 , wherein the pattern is dependent on a device on which the graphical user interface is rendered.

15. The system of claim 1 , wherein the pattern is dependent on a channel on which the sponsored and organic pieces of content are to be displayed.

16. The system of claim 1 , wherein the engagement model is trained using training data regarding user features of a plurality of users and outputs engagement scores based on user features for a user of the graphical user interface.

17. The system of claim 1 , wherein the engagement scores are predicted click-through rates.

18. A computerized method comprising:
accessing a first list of sponsored pieces of content, the first list ordered in a first order;
accessing a second list of organic pieces of content, the second list ordered in a second order;
merging the first list and the second list into a merged list including a plurality of pieces of content;
calculating blended engagement scores for the plurality of pieces of content by:
passing the plurality of pieces of content to an engagement model, the engagement model being a machine-learned model trained by a machine learning algorithm to output an engagement score indicative of a likelihood that a user will engage with a corresponding piece of content if presented with the corresponding piece of content in a user interface; and
combining the engagement scores for the plurality of pieces of content with bids to arrive at blended engagement scores for the plurality of pieces of content;
ranking the plurality of pieces of content using a first ranking algorithm, based on their respective blended engagement scores, resulting in the plurality of pieces of content having organic pieces of content ranked in a third order different than the second order;
creating a pattern for display of the pieces of content by indicating, at each slot in the pattern, whether the corresponding piece of content in the ranking of the plurality of pieces of content is sponsored or organic;
assigning one or more sponsored pieces of content, from the first list to slots in the pattern, that have been identified as sponsored;
assigning one or more organic pieces of content, from the second list to slots in the pattern, that have been identified as organic, the assigning the organic pieces of content in a fourth order different than the third order, the fourth ordering created using a second ranking algorithm different than the first ranking algorithm; and
causing display, in a graphical user interface, of the assigned one or more sponsored pieces of content and the assigned one or more organic pieces of content in the slots in the pattern in which they have been assigned.

19. A system comprising:
means for accessing a first list of sponsored pieces of content, the first list ordered in a first order;
means for accessing a second list of organic pieces of content, the second list ordered in a second order;
means for merging the first list and the second list into a merged list including a plurality of pieces of content;
means for calculating blended engagement scores for the plurality of pieces of content by:
passing the plurality of pieces of content to an engagement model, the engagement model being a machine-learned model trained by a machine learning algorithm to output an engagement score indicative of a likelihood that a user will engage with a corresponding

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Liang Keqing
• Martini Giorgio Paolo
• Zhou Shan
• Fayad Linda
• Pu Wen
• Lu Austin Qingfeng

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06Q10/1053 Primary

  Employment or hiring · CPC title

• G06Q30/02

  Marketing; Price estimation or determination; Fundraising · CPC title

• H04L67/535

  Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34) · CPC title

• H04N21/26258

  for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list (retrieval of multimedia data based on playlists G06F16/40) · CPC title