Contribution incrementality machine learning models

Patent metadata

Field	Value
Publication number	US-11983089-B2
Application number	US-201917278395-A
Country	US
Kind code	B2
Filing date	Dec 5, 2019
Priority date	Dec 5, 2019
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

Methods, systems, and computer programs encoded on a computer storage medium, for training and using machine learning models are disclosed. Methods include creating a model that represents relationships between user attributes, content exposures, and performance levels for a target action using organic exposure data specifying one or more organic exposures experienced by a particular user over a specified time prior to performance of a target action by the particular user and third party exposure data specifying third party exposures of a specified type of digital component to the particular user over the specified time period. Using the model, an incremental performance level attributable to each of the third party exposures at an action time when the target action was performed by the particular user is determined. Transmission criteria for at least some digital components to which the particular user was exposed are modified based on the incremental performance.

First claim

What is claimed is:

1. A method comprising:
creating, by one or more computing devices, a model that represents relationships between user attributes, content exposures, and performance levels for a specified target action;
obtaining organic exposure data specifying one or more organic exposures experienced by a particular user over a specified time period prior to performance of a specified target action by the particular user, wherein the organic exposures are neither exposures to a specified type of digital component nor performance of the specified target action;
obtaining third party exposure data specifying third party exposures of the specified type of digital component to the particular user over the specified time period, wherein the third party exposure data includes, for each of the third party exposures, an exposure time specifying when the third party exposure occurred;
determining, using the model, an incremental performance level attributable to each of the third party exposures at an action time when the specified target action was performed by the particular user;
modifying, by the one or more computing devices, transmission criteria for at least some digital components to which the particular user was exposed based on the incremental performance level that is attributed to the third party exposures of the at least some of the digital components.

2. The method of claim 1 , further comprising:
performing ablation experiments to obtain a set of control outcomes for a set of control users that are not exposed to a particular set of digital components, the control outcomes specifying, for each particular control user in the set of control users, whether the particular control user performed the specified action;
collecting exposure outcomes for a set of exposed users that are not included in the set of control users, the exposure outcomes specifying, for each exposure user in the set of exposure users, whether the exposure user performed the specified action.

3. The method of claim 2 , wherein:
creating the model comprises creating, using a machine learning framework, the model using user attributes, the control outcome for each particular control user, and the exposure outcome for each exposure user.

4. The method of claim 1 wherein determining the incremental performance level attributable to each of the third party exposures comprises:
for each third party exposure:
determining a difference between the exposure time of the third party exposure and the action time when the specified target action occurred;
determining, based on the difference between the exposure time of the third party exposure and the action time when the specified target action occurred, a residual amount of performance level contribution from the third party exposure that remains at the action time; and
attributing the residual amount of the performance level to the third party exposure.

5. The method of claim 4 , further comprising:
determining, for each different type of third party exposure, a decay function specifying a rate of decay of the performance level contribution remaining as a function of time; and
determining for each third party exposure, the residual amount of performance level contribution from the third party exposure that remains at the action time based on the decay function and the difference between the exposure time of the third party exposure and the action time.

6. The method of claim 1 , wherein modifying transmission criteria for at least some digital components comprises adjusting the transmission criteria for a particular digital component in proportion to a magnitude of the incremental performance level attributed to third party exposures of the particular digital component.

7. The method of claim 6 , wherein adjusting the transmission criteria includes disabling a particular transmission criterion having less than a specified magnitude of the incremental performance level attributed to the third party exposures of the particular transmission criterion.

8. A system, comprising:
a data store storing one or more evaluation rules; and
one or more data processors configured to interact with the one or more evaluation rules, and perform operations comprising:
creating, a model that represents relationships between user attributes, content exposures, and performance levels for a specified target action;
obtaining organic exposure data specifying one or more organic exposures experienced by a particular user over a specified time period prior to performance of a specified target action by the particular user, wherein the organic exposures are neither exposures to a specified type of digital component nor performance of the specified target action;
obtaining third party exposure data specifying third party exposures of the specified type of digital component to the particular user over the specified time period, wherein the third party exposure data includes, for each of the third party exposures, an exposure time specifying when the third party exposure occurred;
determining, using the model, an incremental performance level attributable to each of the third party exposures at an action time when the specified target action was performed by the particular user;
modifying, by the one or more processors, transmission criteria for at least some digital components to which the particular user was exposed based on the incremental performance level that is attributed to the third party exposures of the at least some of the digital components.

9. A system of claim 8 , wherein the one or more data processors are configured to perform operations comprising:
performing ablation experiments to obtain a set of control outcomes for a set of control users that are not exposed to a particular set of digital components, the control outcomes specifying, for each particular control user in the set of control users, whether the particular control user performed the specified action;
collecting exposure outcomes for a set of exposed users that are not included in the set of control users, the exposure outcomes specifying, for each exposure user in the set of exposure users, whether the exposure user performed the specified action.

10. The system of claim 9 , wherein:
creating the model comprises creating, using a machine learning framework, the model using user attributes, the control outcome for each particular control user, and the exposure outcome for each exposure user.

11. The system of claim 8 wherein determining the incremental performance level attributable to each of the third party exposures comprises:
for each third party exposure:
determining a difference between the exposure time of the third party exposure and the action time when the specified target action occurred;
determining, based on the difference between the exposure time of the third party exposure and the action time when the specified target action occurred, a residual amount of performance level contribution from the third party exposure that remains at the action time; and
attributing the residual amount of the performance level to the third party exposure.

12. The system of claim 11 , wherein the one or more data processors are configured to perform operations comprising:
determining, for each different type of third party exposure, a decay function specifying a rate of decay of the performance level contribution remaining as a function of time; and
determining for each third party exposure, the residual amount of performance level contribution from the third party exposure that remains at the action time based on the decay function and the difference between the exposure

Assignees

• Google Llc

Inventors

• Bao Xinlong
• Amini Ali Nasiri
• Wang Jing
• Dikmen Mert
• Richardson Amy
• Shender Dinah
• Takagi Junji
• Li Sen
• Jiang Ruoyi
• Jiao Yang
• Zhang Yang
• Zhang Zhuo

Classifications

• G06F11/3433 Primary

  for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029) · CPC title

• G06F11/3428

  Benchmarking · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06Q30/0201 Primary

  Market modelling; Market analysis; Collecting market data · CPC title

• G06Q30/0202

  Market predictions or forecasting for commercial activities · CPC title