Outlier data detection

Patent metadata

Field	Value
Publication number	US-10713683-B2
Application number	US-201514734515-A
Country	US
Kind code	B2
Filing date	Jun 9, 2015
Priority date	Jun 9, 2015
Publication date	Jul 14, 2020
Grant date	Jul 14, 2020

Abstract

Online experimentation has been widely used to evaluate an effect of a new feature of an online product on user engagement. One challenge is that an existence of outliers can often complicate the analysis of such experimental results. Thus, a procedure is provided herein to detect and remove outliers from experimental results. The procedure can use statistical tests based on parametric distributions of sample maximum or minimum. These tests can be performed using an inward testing procedure to identify multiple outliers. Finally, these filtered test results can be used to control delivery of a new feature of an online product.

First claim

The invention claimed is:

1. A system, comprising:
testing circuitry, configured to:
test a feature of an online product for an effect of the feature on user engagement with the online product; and
communicate user engagement data associated with the effect to requesting circuitry;

outlier removal circuitry, configured to:
request the user engagement data from the testing circuitry;
detect an outlier value in the user engagement data;
remove the outlier value in the user engagement data to generate updated user engagement data, wherein the updated user engagement data does not comprise the outlier value;
communicate the updated user engagement data subsequent to removal of the outlier value to the requesting circuitry; and
wherein detecting the outlier value comprises:
estimating parameters of a distribution of the user engagement data using a hurdle model and a negative binomial distribution;
identifying a maximum value or a minimum value in the user engagement data according to the parameters; and
determining whether the maximum value or the minimum value is at least one outlier value according to a distribution of sample maximum or a distribution of sample minimum; and

feature delivery circuitry, configured to:
request at least some user engagement data from the outlier removal circuitry; and
control delivery of one or more features of one or more online products according to the updated user engagement data such that delivery of at least one of the one or more features of the one or more online products is guided by the updated user engagement data generated based upon the removal of the outlier value detected in the user engagement data associated with the effect of the feature, wherein the controlling delivery of the one or more features of the one or more online products comprises (i) formatting a shape of at least one feature of the one or more online products based upon the updated user engagement data generated based upon the removal of the outlier value detected in the user engagement data associated with the effect of the feature, (ii) determining a plurality of spaces on an electronic property of the one or more online products and (iii) matching the shape of the at least one feature to a space, of the plurality of spaces, determined to be available on the electronic property of the one or more online products.

2. The system of claim 1 , wherein the controlling delivery of the one or more features of the online product comprises formatting text of the one or more online products based upon the updated user engagement data generated based upon the removal of the outlier value detected in the user engagement data associated with the effect of the feature.

3. The system of claim 1 , wherein the controlling delivery of the one or more features of the online product comprises placing the at least one feature in a location of the one or more online products based upon the updated user engagement data generated based upon the removal of the outlier value detected in the user engagement data associated with the effect of the feature.

4. The system of claim 1 , wherein the controlling delivery of the one or more features of the online product comprises formatting a graphic of the at least one feature of the one or more online products based upon the updated user engagement data generated based upon the removal of the outlier value detected in the user engagement data associated with the effect of the feature.

5. The system of claim 1 , wherein the controlling delivery of the one or more features of the online product comprises formatting a size of the at least one feature of the one or more online products based upon the updated user engagement data generated based upon the removal of the outlier value detected in the user engagement data associated with the effect of the feature.

6. The system of claim 1 , wherein the outlier removal circuitry is configured to remove the maximum value or the minimum value based on the determination of whether the maximum value or the minimum value is at least one outlier value.

7. The system of claim 6 , wherein the outlier removal circuitry is configured to inward test the user engagement data by repeating the identification, the determination, and the removal of the maximum value or the minimum value until the outlier removal circuitry determines that the maximum value or the minimum value is not at least one outlier value.

8. The system of claim 1 , wherein the effect of the feature on user engagement with the online product is relative to page views of the online product.

9. The system of claim 1 , wherein the effect of the feature on user engagement with the online product is relative to clicks on a part of the online product.

10. The system of claim 1 , wherein the distribution of sample maximum or the distribution of sample minimum is a parametric distribution.

11. The system of claim 1 , wherein the outlier value indicates statistically that more likely than not the outlier value is invalid.

12. The system of claim 1 , wherein the outlier value indicates statistically that more likely than not the outlier value is due to user engagement with the online product by an Internet bot.

13. The system of claim 1 , wherein the outlier value indicates statistically that more likely than not the outlier value is due to fraudulent user engagement with the online product.

14. A method, comprising:
testing, by a processor, a feature of an online product for an effect of the feature on user engagement with the online product, resulting in user engagement data;
estimating parameters of a distribution of the user engagement data using a hurdle model and a negative binomial distribution, the hurdle model generating a binary outcome that indicates zero versus positive values in the user engagement data and the negative binomial distribution based upon the positive values;
identifying a maximum value in the user engagement data according to the parameters;
determining that the maximum value is an outlier value according to a parametric distribution of sample maximum;
transforming the user engagement data by removing the outlier value in the user engagement data to generate updated user engagement data, wherein the updated user engagement data does not comprise the outlier value; and
controlling, by the processor, delivery of one or more features of the online product according to the updated user engagement data such that delivery of at least one of the one or more features of the online product is guided by the updated user engagement data generated based upon the removal of the outlier value determined in the user engagement data associated with the effect of the feature, wherein the controlling delivery of the one or more features of the online product comprises (i) formatting a shape of at least one feature of the online product based upon the updated user engagement data generated based upon the removal of the outlier value determined in the user engagement data associated with the effect of the feature, (ii) determining a plurality of spaces on an electronic property of the online product and (iii) matching the shape of the at least one feature to a space, of the plurality of spaces, determined to be available on the electronic property of the online product.

15. The method of claim 14 , further comprising inward testing the user engagement data by repeating the identification, the determination, and the transformation until determining that the maximum value is not at least one outlier value.

16. The method of claim 14 , wherein the effect of th

Assignees

• Oath Inc

Inventors

• He Yan
• Chen Miao
• Stone Maria

Classifications

• G06Q30/0277

  Online advertisement · CPC title

• G06Q10/04

  Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202) · CPC title

• G06Q30/0243 Primary

  Comparative campaigns · CPC title