Method, device, and non-transitory machine readable medium for content item selection and measurement determination

Patent metadata

Field	Value
Publication number	US-12450650-B2
Application number	US-201916707431-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Dec 9, 2019
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

One or more computing devices, systems, and/or methods for selecting content items for transmission to client devices are provided. A request for content may be received. A second plurality of content items associated with a plurality of rankings may be selected from a plurality of content items based upon a plurality of bid values. A quantity of content items of the second plurality of content items may be greater than a quantity of content items associated with the request for content. One or more ghost content items associated with one or more rankings of the plurality of rankings may be identified. A ghost impression indicator associated with a ghost content item of the one or more ghost content items may be stored based upon a determination that a ranking, of the one or more rankings, associated with the ghost content item matches a second ranking of a presented content item.

First claim

What is claimed is:

1. A method for performing an experiment using a plurality of client devices, comprising:
generating a test group and a control group for the experiment by splitting the plurality of client devices into a first set of client devices associated with the test group and a second set of client devices associated with the control group;
receiving, by one or more servers, a first request for content associated with a first client device of the first set of client devices, wherein the first request for content is indicative of a first quantity of content items;
extracting at least one of a format or a structure of content items from the first request for content;
searching a content item database for content items having at least one of the format or the structure of content items extracted from the first request for content;
retrieving a first plurality of content items from the content item database based upon a determination that content items of the first plurality of content items match at least one of the format or the structure of content items extracted from the first request for content;
determining a first plurality of bid values associated with the first plurality of content items;
selecting, by the one or more servers, a second plurality of content items from the first plurality of content items based upon the first plurality of bid values, wherein:
the second plurality of content items is associated with a first plurality of rankings; and
a second quantity of content items of the second plurality of content items is greater than the first quantity of content items associated with the first request for content;

extracting one or more experiment statuses associated with the second plurality of content items;
identifying, by the one or more servers, one or more first ghost content items of the second plurality of content items based upon a determination that the one or more first ghost content items are each associated with an experiment status indicative of an ongoing experiment, wherein the one or more first ghost content items are associated with one or more first rankings of the first plurality of rankings;
selecting, by the one or more servers, a third plurality of content items, of the second plurality of content items, that are not ghost content items for presentation via the first client device of the test group of the experiment, wherein:
a third quantity of content items of the third plurality of content items is equal to the first quantity of content items associated with the first request for content; and
the third plurality of content items is associated with a second plurality of rankings;

providing, by the one or more servers, a first content item, of the third plurality of content items that are not ghost content items, to the first client device of the test group of the experiment, wherein:
the first content item is associated with a second ranking of the second plurality of rankings;
a second content item, of the third plurality of content items, is associated with a third ranking of the second plurality of rankings; and
each content item is displayed in a serve area on the first client device corresponding to a rank of the content item;

providing, by the one or more servers, a first ghost content item, of the one or more first ghost content items, to the second set of client devices of the control group of the experiment;
determining whether a first ranking associated with the first ghost content item, matches one or more rankings associated with one or more presented content items;
responsive to determining that at least a threshold proportion of the first ghost content item was displayed on the first client device, generating and storing, by the one or more servers, a first ghost impression indicator associated with the first ghost content item based upon a determination that the first ranking associated with the first ghost content item matches the second ranking associated with the first content item presented via the first client device;
after storing the first ghost impression indicator, determining, by the one or more servers, one or more measurements associated with the first ghost content item;
tracking, by the one or more servers, at least some of the plurality of client devices that presented one or more content items;
based upon the tracking the plurality of client devices, retrieving at least one of one or more instant messages or one or more emails associated with one or more devices of at least one of the first set of client devices comprising the first client device or the second set of client devices that at least one of received or presented the first ghost content item;
analyzing at least one of the one or more instant messages or the one or more emails to determine (i) first activity associated with the first set of client devices associated with the test group and (ii) second activity associated with the second set of client devices, associated with the control group, that at least one of received or presented the first ghost content item;
determining a first relationship between (i) the first activity associated with the first set of client devices comprising the first client device and (i) the second activity associated with the second set of client devices that at least one of received or presented the first ghost content item;
modifying, based upon the first relationship, a transmission control data structure associated with the first ghost content item; and
controlling, by the one or more servers and based upon the modified transmission control data structure, transmission of one or more content items to one or more client devices, wherein the controlling comprises increasing a rate of transmission to client devices associated with first characteristics and decreasing a rate of transmission to client devices associated with second characteristics based upon the one or more measurements, wherein the controlling creates a closed-loop process that uses results of the experiment as feedback to tailor one or more parameters of a content system.

2. The method of claim 1 , comprising:
prior to the storing the first ghost impression indicator, receiving impression information indicative of the first content item being presented via the first client device, wherein:
the storing the first ghost impression indicator is performed responsive to the receiving the impression information indicative of the first content item being presented via the first client device.

3. The method of claim 1 , comprising:
analyzing content information associated with the second plurality of content items to identify one or more potential ghost content items, wherein each potential ghost content item of the one or more potential ghost content items is associated with a group of client devices to which the potential ghost content item is not provided; and
analyzing ghost information associated with the one or more potential ghost content items to determine that, for each ghost content item of the one or more first ghost content items, the first client device is included in a group of client devices to which the ghost content item is not provided, wherein the identifying the one or more first ghost content items is performed based upon the determination that, for each ghost content item of the one or more first ghost content items, the first client device is included in a group of client devices to which the ghost content item is not provided.

4. The method of claim 1 , comprising:
identifying a fourth plurality of content items of the second plurality of content items that are not the one or more first ghost content items; and
determining a third plurality of rankings associated with the fourth plurality of content items based upon the first plurality of rankings, wh

Assignees

• Oath Inc
• Yahoo Ad Tech Llc

Inventors

• Barajas Zamora Joel
• Bhamidipati Lakshmi Narayan
• Slain Ilya Nathaniel
• Medina Lioz
• Krasteva Milena Petrova

Classifications

• G06Q30/0275

  Auctions · CPC title

• G06Q30/0241

  Advertisements · CPC title

• G06Q30/0631 Primary

  Recommending goods or services · CPC title

• G06Q30/08 Primary

  Auctions · CPC title