Contribution incrementality machine learning models

Patent metadata

Field	Value
Publication number	US-12481571-B2
Application number	US-202418625830-A
Country	US
Kind code	B2
Filing date	Apr 3, 2024
Priority date	Dec 5, 2019
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

Methods, systems, and computer programs encoded on a computer storage medium, for training and using machine learning models are disclosed. Methods include creating a model that represents relationships between user attributes, content exposures, and performance levels for a target action using organic exposure data specifying one or more organic exposures experienced by a particular user over a specified time prior to performance of a target action by the particular user and third party exposure data specifying third party exposures of a specified type of digital component to the particular user over the specified time period. Using the model, an incremental performance level attributable to each of the third party exposures at an action time when the target action was performed by the particular user is determined. Transmission criteria for at least some digital components to which the particular user was exposed are modified based on the incremental performance.

First claim

What is claimed is:

1 . A method comprising:
creating, by one or more computing devices, a model that represents relationships between user attributes, content exposures, and performance levels for a specified target action; obtaining content exposure data specifying content exposures experienced by a particular user; determining, using the model and the content exposure data, an incremental performance levels attributable to each of the content exposures at an action time when the specified target action was performed by the particular user; and modifying, by the one or more computing devices, transmission criteria for at least some digital components to which the particular user was exposed based on the incremental performance levels.

2 . The method of claim 1 , further comprising:
performing ablation experiments to obtain a set of control outcomes for a set of control users that are not exposed to a particular set of digital components, the control outcomes specifying, for each particular control user in the set of control users, whether the particular control user performed the specified action; collecting exposure outcomes for a set of exposed users that are not included in the set of control users, the exposure outcomes specifying, for each exposure user in the set of exposure users, whether the exposure user performed the specified action.

3 . The method of claim 2 , wherein:
creating the model comprises creating, using a machine learning framework, the model using user attributes, the control outcome for each particular control user, and the exposure outcome for each exposure user.

4 . The method of claim 1 , wherein:
the content exposures include third party exposures; and determining the incremental performance level comprises:
for each third party exposure among the content exposures:
determining a difference between the exposure time of the third party exposure and the action time when the specified target action occurred;
determining, based on the difference between the exposure time of the third party exposure and the action time when the specified target action occurred, a residual amount of performance level contribution from the third party exposure that remains at the action time; and
attributing the residual amount of the performance level to the third party exposure.

5 . The method of claim 4 , further comprising:
determining, for each different type of third party exposure, a decay function specifying a rate of decay of the performance level contribution remaining as a function of time; and determining for each third party exposure, the residual amount of performance level contribution from the third party exposure that remains at the action time based on the decay function and the difference between the exposure time of the third party exposure and the action time.

6 . The method of claim 4 , wherein modifying transmission criteria for at least some digital components comprises adjusting the transmission criteria for a particular digital component in proportion to a magnitude of the incremental performance level attributed to third party exposures of the particular digital component.

7 . The method of claim 6 , wherein adjusting the transmission criteria includes disabling a particular transmission criterion having less than a specified magnitude of the incremental performance level attributed to the third party exposures of the particular transmission criterion.

8 . A system, comprising:
a data store storing one or more evaluation rules; and one or more data processors configured to interact with the one or more evaluation rules, and perform operations comprising:
creating, by one or more computing devices, a model that represents relationships between user attributes, content exposures, and performance levels for a specified target action;
obtaining content exposure data specifying content exposures experienced by a particular user;
determining, using the model and the content exposure data, an incremental performance levels attributable to each of the content exposures at an action time when the specified target action was performed by the particular user; and
modifying, by the one or more computing devices, transmission criteria for at least some digital components to which the particular user was exposed based on the incremental performance levels.

9 . The system of claim 8 , wherein the one or more data processors are configured to perform operations comprising:
performing ablation experiments to obtain a set of control outcomes for a set of control users that are not exposed to a particular set of digital components, the control outcomes specifying, for each particular control user in the set of control users, whether the particular control user performed the specified action; collecting exposure outcomes for a set of exposed users that are not included in the set of control users, the exposure outcomes specifying, for each exposure user in the set of exposure users, whether the exposure user performed the specified action.

10 . The system of claim 9 , wherein:
creating the model comprises creating, using a machine learning framework, the model using user attributes, the control outcome for each particular control user, and the exposure outcome for each exposure user.

11 . The system of claim 8 , wherein;
the content exposures include third party exposures; and determining the incremental performance level comprises:
for each third party exposure among the content exposures:
determining a difference between the exposure time of the third party exposure and the action time when the specified target action occurred;
determining, based on the difference between the exposure time of the third party exposure and the action time when the specified target action occurred, a residual amount of performance level contribution from the third party exposure that remains at the action time; and
attributing the residual amount of the performance level to the third party exposure.

12 . The system of claim 11 , wherein the one or more data processors are configured to perform operations comprising:
determining, for each different type of third party exposure, a decay function specifying a rate of decay of the performance level contribution remaining as a function of time; and determining for each third party exposure, the residual amount of performance level contribution from the third party exposure that remains at the action time based on the decay function and the difference between the exposure time of the third party exposure and the action time.

13 . The system of claim 11 , wherein modifying transmission criteria for at least some digital components comprises adjusting the transmission criteria for a particular digital component in proportion to a magnitude of the incremental performance level attributed to third party exposures of the particular digital component.

14 . The system of claim 13 , wherein adjusting the transmission criteria includes disabling a particular transmission criterion having less than a specified magnitude of the incremental performance level attributed to the third party exposures of the particular transmission criterion.

15 . A non-transitory computer readable medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations comprising:
creating, by one or more computing devices, a model that represents relationships between user attributes, content exposures, and performance levels for a specified targ

Assignees

• Google Llc

Inventors

• Bao Xinlong
• Nasiri Amini Ali
• Wang Jing
• Dikmen Mert
• Richardson Amy
• Shender Dinah
• Takagi Junji
• Li Sen
• Jiang Ruoyi
• Jiao Yang
• Zhang Yang
• Zhang Zhuo

Classifications

• G06F11/3428

  Benchmarking · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• G06Q30/0202

  Market predictions or forecasting for commercial activities · CPC title

• G06F11/3433 Primary

  for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029) · CPC title