Online product testing using bucket tests

Patent metadata

Field	Value
Publication number	US-2016103758-A1
Application number	US-201414509741-A
Country	US
Kind code	A1
Filing date	Oct 8, 2014
Priority date	Oct 8, 2014
Publication date	Apr 14, 2016
Grant date	—

Abstract

The technologies described herein use a statistical test to determine whether differences between data sets of buckets in a bucket test, such as differences between averages of two buckets (e.g., differences between means of two buckets), are directionally larger than a predetermined or preset minimum threshold value. The statistical test may also provide an extension to specify the minimum threshold value as a percentage. Also, described herein are techniques for estimating different control variables of a bucket test, such as estimating minimum bucket size to provide sufficient statistical power with use of the minimum threshold value.

First claim

1 . Testing circuitry for bucket testing, comprising:
threshold metric circuitry configured to store a threshold metric of a bucket test of an update to an online product, wherein the threshold metric includes a software metric associated with the online product; minimum difference circuitry configured to store a predetermined minimum difference of the threshold metric; and confidence circuitry configured to store a confidence interval, a p-value, a test conclusion, or any combination thereof of the threshold metric.

2 . The testing circuitry of claim 1 , further comprising control circuitry configured to store a control metric of the bucket test.

3 . The testing circuitry of claim 2 , further comprising launch circuitry configured to provide the update to the online product where the test conclusion indicates that with pre-specified confidence a resulting difference of the bucket test is greater than the predetermined minimum difference.

4 . The testing circuitry of claim 2 , further comprising test-running circuitry configured to run the bucket test according to the control metric.

5 . The testing circuitry of claim 2 , wherein the control metric is a bucket size of the bucket test.

6 . The testing circuitry of claim 2 , wherein the control metric is a time period of the bucket test.

7 . The testing circuitry of claim 1 , wherein the bucket test includes an A/B test.

8 . The testing circuitry of claim 1 , wherein the threshold metric is a primary metric, wherein the software metric is a primary software metric, wherein the testing circuitry further comprises non-threshold metric circuitry configured to store a secondary metric, and wherein the secondary metric is a secondary software metric.

9 . The testing circuitry of claim 8 , further comprising secondary difference circuitry configured to store a difference associated with the secondary metric.

10 . The testing circuitry of claim 8 , further comprising control circuitry configured to store a control metric of the bucket test associated with the secondary metric.

11 . The testing circuitry of claim 10 , wherein the bucket test is a first bucket test, and wherein the control metric is a bucket size of a second bucket test associated with the secondary metric.

12 . The testing circuitry of claim 10 , wherein the control metric is a time period of the bucket test.

13 . The testing circuitry of claim 1 , further comprising a graphical user interface (GUI), and wherein the GUI includes respective fields configured to display the threshold metric, the predetermined minimum difference, the confidence interval, the p-value, the test conclusion, or any combination thereof.

14 . The testing circuitry of claim 13 , wherein the GUI includes a dashboard, and wherein the respective fields update in real time during the bucket test.

15 . The testing circuitry of claim 13 , further comprising metric generation circuitry configured to generate an additional metric, and wherein the GUI further includes a graphical field configured to initiate the generation of the additional metric.

16 . A method, comprising:
storing, in threshold metric circuitry, a threshold metric of a bucket test of an update to an online product, wherein the threshold metric includes a software metric associated with the online product; storing, in minimum difference circuitry, a predetermined minimum difference of the threshold metric; storing, in control circuitry, a control metric; running, by test-running circuitry, a one-sided bucket test using the threshold metric, the predetermine minimum difference, and the control metric, which results in a test conclusion; storing, by confidence circuitry, the test conclusion; and providing, by launch circuitry, the update to the online product.

17 . The method of claim 16 , wherein the control metric is a bucket size of the bucket test.

18 . The method of claim 16 , wherein the control metric is a time period of the bucket test.

19 . A method, comprising:
selecting, by bucket testing circuitry, a primary attribute according to analytics; determining, by the circuitry, whether to consider a secondary attribute according to the analytics; selecting, by the circuitry, a one-sided test with a minimum difference for the primary attribute according to the determination of whether to consider the secondary attribute; and running, by the circuitry, the one-sided test with the minimum difference using the primary attribute as a threshold metric.

20 . The method of claim 19 , further comprising:
selecting, by the circuitry, the secondary attribute according to the analytics; selecting, by the circuitry, a standard one-sided test or a standard two-sided test for the secondary attribute; and running, by the circuitry, the standard one-sided test or the standard two-sided test accordingly, using the secondary attribute as a non-threshold metric.

Assignees

• Yahoo Inc

Inventors

• Zhao Zhenyu
• Oliveira Flavio T P
• Stone Maria
• Chen Miao
• Pandey Shalu
• Tripathi Kshitiz

Classifications

• G06F11/3684 Primary

  for test design, e.g. generating new test cases · CPC title

• G06F11/3672

  Test management · CPC title

• G06F11/3616

  using software metrics · CPC title

• G06F11/3664 Primary

  Physics · mapped topic

• G06F8/65 Primary

  Updates (security arrangements therefor G06F21/57) · CPC title