Causal analysis system

Patent metadata

Field	Value
Publication number	US-11468348-B1
Application number	US-202016788100-A
Country	US
Kind code	B1
Filing date	Feb 11, 2020
Priority date	Feb 11, 2020
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

Methods and apparatus for identifying features that may have a high potential impact on key application metrics. These methods rely on observational data to estimate the importance of application features, and use causal inference tools such as Double Machine Learning (double ML) or Recurrent Neural Networks (RNN) to estimate the impacts of treatment features on key metrics. These methods may allow developers to estimate the effectiveness of features without running online experiments. These methods may, for example, be used to effectively plan and prioritize online experiments. Results of the online experiments may be used to optimize key metrics of mobile applications, web applications, websites, and other web-based programs.

First claim

What is claimed is:

1. A system, comprising;
one or more processors; and
memory comprising program instructions that when executed on the one or more processors implement a causal inference engine configured to:
receive treatment feature data from a time period t−1 for one or more treatment features of an application, wherein the treatment features include features of interest in regard to a target metric of the application in a time period t;
receive control feature data for one or more control features of the application, wherein the control features include information about user behavior over a period of time and are not in the treatment features;
perform a causal analysis of the treatment features with respect to the target metric for the time period t−1 using the treatment feature data and the control feature data to generate prioritized treatment features with respect to estimated impact on the target metric; and
output the prioritized treatment features to an application testing process, wherein the application testing process prioritizes online testing of the application based on the prioritized treatment features to determine one or more of the treatment features that have an impact on the target metric that is above a specified threshold.

2. The system as recited in claim 1 , wherein, to perform a causal analysis of the treatment features with respect to the target metric, the causal inference engine is configured to input the treatment feature data and the control feature data to a trained machine learning (ML) model.

3. The system as recited in claim 2 , wherein the ML model is one of a double ML model or a Recurrent Neural Network (RNN) model.

4. The system as recited in claim 1 , wherein the causal inference engine is further configured to perform feature selection on the treatment features prior to performing the causal analysis to reduce the number of treatment features based on correlations between the treatment features and correlations between the treatment features and the target metric.

5. The system as recited in claim 1 , wherein the causal inference engine is configured to generate prioritized treatment features for two or more different categories of users.

6. The system as recited in claim 1 , wherein the prioritized treatment features include indications of increase in usage of the treatment features with regard to the target metric and indications of global lift of the treatment features with regard to the target metric.

7. A method, comprising:
performing, by a causal analysis system implemented by one or more computing devices:
receiving treatment features and a target metric of an application in a time period t;
performing a causal analysis of the treatment features with respect to the target metric for a time period t−1 using the treatment features and control features for a time period prior to and including t−1 to generate prioritized treatment features with respect to estimated impact on the target metric; and
prioritizing online testing of the application based on the prioritized treatment features to determine one or more of the treatment features that have a higher impact on the target metric than the other treatment features.

8. The method as recited in claim 7 , wherein the control features include information about user behavior over a prior period of time and are not in the treatment features.

9. The method as recited in claim 7 , wherein performing a causal analysis of the treatment features with respect to the target metric comprises inputting the treatment feature data and the control feature data to a trained machine learning (ML) model.

10. The method as recited in claim 9 , wherein the ML model is a double ML model.

11. The method as recited in claim 10 , wherein performing the causal analysis of a given treatment feature with respect to the target metric using the double ML model comprises:
predicting the target metric using the control features;
predicting the treatment feature using the control features;
determining difference between an actual target metric and the predicted target metric;
determining difference between an actual treatment feature and the predicted treatment feature; and
performing a regression of the determined differences to obtain causal impact of the treatment feature on the target metric.

12. The method as recited in claim 9 , wherein the ML model is Recurrent Neural Network (RNN) model.

13. The method as recited in claim 12 , wherein performing the causal analysis of a given treatment feature with respect to the target metric using the RNN model comprises:
performing a first prediction using all of the features;
performing a second prediction with the given treatment feature zeroed out; and
determining difference between output of the first prediction and the second prediction to obtain causal impact of the treatment feature on the target metric.

14. The method as recited in claim 7 , further comprising performing feature selection on the treatment features prior to performing the causal analysis to reduce the number of treatment features based on correlations between the treatment features and correlations between the treatment features and the target metric.

15. The method as recited in claim 7 , further comprising generating prioritized treatment features for two or more different categories of users.

16. The method as recited in claim 7 , wherein the prioritized treatment features include indications of increase in usage of the treatment features with regard to the target metric and indications of global lift of the treatment features with regard to the target metric.

17. The method as recited in claim 7 , further comprising performing the causal analysis of the treatment features with respect to the target metric for one or more subsequent time periods to monitor impact of the treatment features on the target metric over time.

18. One or more non-transitory computer-readable storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:
obtain treatment features and a target metric of an application in a time period t; and
perform a causal analysis of the treatment features with respect to the target metric for a time period t−1 using the treatment features and control features for a time period prior to and including t−1 to generate prioritized treatment features with respect to estimated impact on the target metric, wherein the control features include information about user behavior over a prior period of time and are not in the treatment features; and
prioritize online testing of the application based on the prioritized treatment features to determine one or more of the treatment features that have a higher impact on the target metric than the other treatment features.

19. The one or more non-transitory computer-readable storage media as recited in claim 18 , wherein, to perform a causal analysis of the treatment features with respect to the target metric, the program instructions when executed on or across the one or more processors further cause the one or more processors to input the treatment feature data and the control feature data to a trained machine learning (ML) model.

20. The one or more non-transitory computer-readable storage media as recited in claim 19 , wherein the ML model is one of a double ML model or a Recurrent Neural Network (RNN) model.

21. The one or more non-transitory computer-readable storage medi

Assignees

• Amazon Tech Inc

Inventors

• Cui Can
• Chatzipanagiotis Nikolaos
• Kuila Tamal Krishna
• Sadagopan Narayanan

Classifications

• G06F18/211

  Selection of the most significant subset of features · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06N3/09

  Supervised learning · CPC title