Training and utilizing multi-phase learning models to provide digital content to client devices in a real-time digital bidding environment

Patent metadata

Field	Value
Publication number	US-11288709-B2
Application number	US-201815938449-A
Country	US
Kind code	B2
Filing date	Mar 28, 2018
Priority date	Mar 28, 2018
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

The present disclosure includes systems, methods, and non-transitory computer readable media that train and utilize multi-phase learning models to predict performance during digital content campaigns and provide digital content to client devices in a real-time bidding environment. In particular, one or more embodiments leverage organizational structure of digital content campaigns to train two learning models, utilizing different data sources, to predict performance, generate bid responses, and provide digital content to client devices. For example, the disclosed systems can train a first performance learning model in an offline mode utilizing parent-level historical data. Then, in an online mode, the disclosed systems can train a second performance learning model utilizing child-level historical data and utilize the first performance learning model and the second performance learning model to generate bid responses and bid amounts in a real-time bidding environment.

First claim

We claim:

1. In a digital medium environment for real-time bidding on impression opportunities corresponding to client computing devices of users navigating to digital assets, a method for utilizing multi-phase learning models to generate bid responses and provide digital assets to select client devices, the method comprising:
identifying, utilizing at least one server device, one or more digital content campaigns comprising a parent keyword and a child keyword, wherein the child keyword comprises an example of the parent keyword;
generating a high-complexity performance learning model, via the at least one server device in an offline mode prior to receiving a request corresponding to an impression opportunity of a client device associated with the child keyword, based on parent-level historical data corresponding to the parent keyword by utilizing the parent-level historical data to tune weights of the high-complexity performance learning model at a first frequency; and
in response to receiving the request corresponding to the impression opportunity of the client device associated with the child keyword, utilizing the at least one server to:
identify a low-complexity performance learning model corresponding to the child keyword generated based on child-level historical data corresponding to the child keyword by determining one or more averages across the child-level historical data, the child-level historical data having a sparse population relative to the parent-level historical data;
identify, in an online mode and after generating the high-complexity performance learning model, updated child-level data corresponding to the child keyword;
update, in the online mode and at a second frequency higher than the first frequency, the low-complexity performance learning model based on the updated child-level historical data corresponding to the child keyword;
utilize the updated low-complexity performance learning model to determine a child-level performance metric for the impression opportunity of the client device associated with the child keyword;
utilize the high-complexity performance learning model to determine a parent-level performance metric for the impression opportunity of the client device associated with the child keyword;
utilize the child-level performance metric determined by the updated low-complexity performance learning model and the parent-level performance metric determined by the high-complexity performance learning model to determine a performance metric corresponding to the child keyword; and
transmit, utilizing the performance metric corresponding to the child keyword, a response comprising a digital bid corresponding to the impression opportunity of the client device.

2. The method as recited in claim 1 , further comprising, in response to receiving an additional request corresponding to an additional impression opportunity of an additional client device associated with the child keyword, utilizing the at least one server to:
identify further updated child-level data corresponding to the child keyword; and
modify the updated low-complexity performance learning model utilizing the further updated child-level data corresponding to the child keyword.

3. The method as recited in claim 2 , further comprising:
determining, utilizing the modified low-complexity performance learning model, an additional child-level performance metric for the additional impression opportunity of the additional child device; and
utilizing the additional child-level performance metric and one or more parent-level performance metrics determined by the high-complexity performance learning model to determine an additional performance metric corresponding to the child keyword.

4. The method as recited in claim 1 , further comprising:
storing, in the offline mode, the tuned weights of the high-complexity performance learning model;
accessing, in the online mode, the tuned weights of the high-complexity performance learning model; and
utilizing the tuned weights of the high-complexity performance learning model, in the online mode, to determine the parent-level performance metric.

5. The method as recited in claim 1 , further comprising generating the high-complexity performance learning model by training a performance neural network utilizing the parent-level historical data by back-propagating to determine one or more line weights associated with nodes of the performance neural network.

6. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, cause a computing device to:
identify one or more digital content campaigns comprising a parent parameter and a child parameter, wherein the parent parameter subsumes the child parameter;
generate, in an offline mode prior to receiving a request corresponding to an impression opportunity of a client device associated with a child parameter, a high-complexity performance learning model at a first frequency based on parent-level historical data corresponding to the parent parameter by utilizing the parent-level historical data to tune weights of the high-complexity performance learning model; and
in response to receiving the request corresponding to the impression opportunity of the client device associated with the child parameter:
identify a low-complexity performance learning model corresponding to the child parameter generated based on child-level historical data, the child-level historical data having a sparse population relative to the parent-level historical data;
identify, in an online mode and after generating the high-complexity performance learning model, updated child-level data corresponding to the child parameter;
update, in the online mode and at a second frequency higher than the first frequency, the low-complexity performance learning model based on the updated child-level historical data associated with the child parameter;
utilize the updated low-complexity performance learning model to determine a child-level performance metric for the impression opportunity of the client device associated with the child parameter;
utilize the high-complexity performance learning model to determine parent-level performance metric for the impression opportunity of the client device associated with the child parameter;
utilize the child-level performance metric determined by the updated low-complexity performance learning model and the parent-level performance metric determined by the high-complexity performance learning model to determine a performance metric corresponding to the child parameter; and
transmit, utilizing the performance metric corresponding to the child parameter, a response comprising a digital bid corresponding to the impression opportunity of the client device.

7. The non-transitory computer-readable medium as recited in claim 6 , wherein the parent parameter comprises a parent keyword, the child parameter comprises a child keyword of the parent keyword, and the impression opportunity corresponds to a search query utilizing the child keyword.

8. The non-transitory computer-readable medium as recited in claim 6 , further comprising computer-executable instructions that, when executed by the processor, cause the computing device to generate the high-complexity performance learning model by training at least one of a linear regression model or a neural network.

9. The non-transitory computer-readable medium as recited in claim 8 , further comprising computer-executable instructions that, when executed by the processor, cause the computing device to:
generate the low-complexity performance learning model utilizing the child-level historical data;
after generating the low-complexit

Assignees

• Adobe Inc

Inventors

• Yan Zhenyu
• Dong Chen
• Pani Abhishek
• Yuan Yuan

Classifications

• G06Q30/0275 Primary

  Auctions · CPC title

• G06Q30/0242 Primary

  Determining effectiveness of advertisements · CPC title

• G06N20/00

  Machine learning · CPC title