Risk Quantification for Policy Deployment

Patent metadata

Field	Value
Publication number	US-2016148251-A1
Application number	US-201414552047-A
Country	US
Kind code	A1
Filing date	Nov 24, 2014
Priority date	Nov 24, 2014
Publication date	May 26, 2016
Grant date	—

Abstract

Risk quantification, policy search, and automated safe policy deployment techniques are described. In one or more implementations, techniques are utilized to determine safety of a policy, such as to express a level of confidence that a new policy will exhibit an increased measure of performance (e.g., interactions or conversions) over a currently deployed policy. In order to make this determination, reinforcement learning and concentration inequalities are utilized, which generate and bound confidence values regarding the measurement of performance of the policy and thus provide a statistical guarantee of this performance. These techniques are usable to quantify risk in deployment of a policy, select a policy for deployment based on estimated performance and a confidence level in this estimate (e.g., which may include use of a policy space to reduce an amount of data processed), used to create a new policy through iteration in which parameters of a policy are iteratively adjusted and an effect of those adjustments are evaluated, and so forth.

First claim

What is claimed is:

1 . In a digital medium environment for identifying and deploying potential digital advertising campaigns, where campaigns can be altered, removed, or replaced on demand, a method for optimizing campaign selection in the digital medium environment, the method comprising:
receiving a policy at one or more computing devices, the policy configured for deployment by a content provider to select advertisements; and controlling deployment, by the one or more computing devices, of the received policy by the content provider based at least in part on a quantification of risk that is likely involved in the deployment of the received policy as opposed to a deployed policy of the content provider, the controlling including:
applying reinforcement learning and a concentration inequality on deployment data that describes the deployment of the deployed policy by the content provider to estimate values of a measure of performance of the received policy and to quantify the risk by calculating one or more statistical guarantees of the estimated values; and
causing deployment of the received policy responsive to a determination that the one or more statistical guarantees express at least a confidence level that the estimated values of the measure of performance at least correspond to a threshold that is based at least in part on a measure of performance of the deployed policy by the content provider.

2 . A method as described in claim 1 , wherein the threshold is based at least in part on the measured performance of the deployed policy and a set margin.

3 . A method as described in claim 2 , wherein the threshold is set such that the estimated values of the received policy exhibit an improvement in the measurement of performance over the deployed policy.

4 . A method as described in claim 1 , wherein the confidence level and the threshold are user definable via interaction with a user interface of the one or more computing devices.

5 . A method as described in claim 1 , wherein the concentration inequality is configured to move estimated values above a defined threshold to lie on the defined threshold.

6 . A method as described in claim 1 , wherein the concentration inequality is configured to be independent of a range of random variables of the estimated values.

7 . A method as described in claim 1 , wherein the concentration inequality is configured to collapse tails of random variable distributions of the estimated values, normalize the random variable distributions, and then generate a lower-bound from which a lower-bound on a uniform mean of original random variables of the estimated values is extracted.

8 . A method as described in claim 1 , wherein the policy is configured for use by the content provider to select advertisements for inclusion with content based at least in part based on characteristics associated with a request to access the content.

9 . A method as described in claim 8 , wherein the characteristics associated with the request include characteristics of a user or device that initiated the request or characteristics of the request itself.

10 . A method as described in claim 8 , wherein the characteristics are expressed using a feature vector.

11 . A method as described in claim 1 , wherein received deployment data does not describe deployment of the received policy by the one or more entities.

12 . A method as described in claim 1 , wherein received deployment data also describes deployment of the received policy.

13 . A system comprising:
one or more computing devices configured to perform operations including controlling deployment of a received policy based at least in part on a quantification of risk that is likely involved in the deployment of the received policy as opposed to a deployed policy, the controlling including:
using reinforcement learning and a concentration inequality on deployment data that describes the deployment of the deployed policy to estimate values of a measure of performance of the received policy and to quantify the risk by calculating one or more statistical guarantees regarding the estimated values; and
causing replacement of the deployment of the deployed policy with the received policy responsive to a determination that the one or more statistical guarantees express at least a confidence level that the estimated values of the measure of performance at least correspond to a threshold that is based at least in part on a measure of performance of the deployed policy.

14 . A system as described in claim 13 , wherein the threshold is based at least in part on the measured performance of the deployed policy and a set margin.

15 . A system as described in claim 14 , wherein the threshold is set such that the estimated values of the received policy exhibit an improvement in the measurement of performance over the deployed policy.

16 . A system as described in claim 13 , wherein the confidence level and the threshold are user definable via interaction with a user interface of the one or more computing devices.

17 . A content provider comprising one or more computing devices configured to perform operations including:
deploying a policy to select advertisements to be included with content based on one or more characteristics associated with a request for the content; and replacing the deployed policy with another policy that is selected through use of reinforcement learning and a concentration inequality to process deployment data and determine that the one or more statistical guarantees express at least a confidence level that estimated values of a measure of performance of the received policy at least corresponds to a threshold that is based at least in part on a measure of performance of the deployed policy.

18 . A content provider as described in claim 17 , wherein the threshold is based at least in part on the measure of performance of the deployed policy and a set margin.

19 . A content provider as described in claim 17 , wherein the threshold is set such that the estimated values of the received policy exhibit an improvement in the measurement of performance over the deployed policy.

20 . A content provider as described in claim 17 , wherein the confidence level and the threshold are user definable via interaction with a user interface of the one or more computing devices.

Assignees

• Adobe Systems Inc

Inventors

• Thomas Philip S
• Theocharous Georgios
• Ghavamzadeh Mohammad

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06Q10/06375

  Prediction of business process outcome or impact based on a proposed change · CPC title

• G06Q30/0243

  Comparative campaigns · CPC title

• G06Q30/0244 Primary

  Optimization · CPC title

• G06N20/00

  Machine learning · CPC title