Orthogonal experimentation in a computing environment

Patent metadata

Field	Value
Publication number	US-10599557-B2
Application number	US-201916292804-A
Country	US
Kind code	B2
Filing date	Mar 5, 2019
Priority date	Mar 19, 2010
Publication date	Mar 24, 2020
Grant date	Mar 24, 2020

Abstract

Various embodiments include at least one of systems, methods, and software to receive input configuring tests within a computing environment to expose users to standard application or website experiences or test experiences. In some embodiments, multiple tests may be configured to run orthogonally within user experiences without affecting the results of one another. Some such embodiments preserve the ability to execute certain tests in a non-orthogonal manner while other tests are allowed to execute orthogonally.

First claim

What is claimed is:

1. A method comprising:
receiving a first experiment identifier for a first experiment;
receiving a first traffic configuration setting for the first experiment, the first traffic configuration setting indicating a portion of traffic allotted to the first experiment;
receiving a first traffic allotment for a first experiment variant of the first experiment and a second traffic allotment for a second experiment variant of the first experiment;
receiving a field value indicating that traffic to the first experiment is mutually excluded from other traffic to a second experiment; and
storing the first experiment identifier, the first traffic configuration setting, the first traffic allotment, the second traffic allotment in an experiment configuration file.

2. The method of claim 1 , further comprising:
performing a hashing function on data including a user identifier to obtain a hash value, the hash value used to determine whether to return a first web page corresponding to the first experiment variant or a second web page corresponding to the second experiment variant.

3. The method of claim 2 , wherein the data further includes the first experimental identifier.

4. The method of claim 3 , further comprising mapping the hash value to a bucket range to determine whether to return a web page corresponding to the first experiment variant or the second experiment variant.

5. The method of claim 2 , further comprising assigning a value to a user corresponding to the user identifier.

6. The method of claim 1 , further comprising receiving second data indicating that traffic to the second experiment is not mutually excluded from traffic to a third experiment.

7. The method of claim 1 , further comprising generating a user interface comprising at least one field for input of the field value indicating that traffic to the first experiment is mutually excluded from other traffic to a second experiment.

8. A system comprising:
at least one processor; and
a memory device storing instructions that, when executed by the at least one processor, cause the system perform operations comprising:
receiving a first experiment identifier for a first experiment;
receiving a first traffic configuration setting for the first experiment, the first traffic configuration setting indicating a portion of traffic allotted to the first experiment;
receiving a first traffic allotment for a first experiment variant of the first experiment and a second traffic allotment for a second experiment variant of the first experiment;
receiving a field value indicating that traffic to the first experiment is mutually excluded from other traffic to a second experiment; and
storing the first experiment identifier, the first traffic configuration setting, the first traffic allotment, the second traffic allotment in an experiment configuration file.

9. The system of claim 8 , wherein the operations further comprise:
performing a hashing function on data including a user identifier to obtain a hash value, the hash value used to determine whether to return a first web page corresponding to the first experiment variant or a second web page corresponding to the second experiment variant.

10. The system of claim 9 , wherein the data further includes the first experimental identifier.

11. The system of claim 10 , wherein the operations further comprise mapping the hash value to a bucket range to determine whether to return a web page corresponding to the first experiment variant or the second experiment variant.

12. The system of claim 9 , wherein the operations further comprise assigning a value to a user corresponding to the user identifier.

13. The system of claim 8 , wherein the operations further comprise receiving second data indicating that traffic to the second experiment is not mutually excluded from traffic to a third experiment.

14. The system of claim 8 , wherein the operations further comprise generating a user interface comprising at least one field for input of the field value indicating that traffic to the first experiment is mutually excluded from other traffic to a second experiment.

15. A non-transitory computer-readable storage medium, with instructions stored thereon, which when executed by at least one processor of a computer, causes the computer to perform operations comprising:
receiving a first experiment identifier for a first experiment;
receiving a first traffic configuration setting for the first experiment, the first traffic configuration setting indicating a portion of traffic allotted to the first experiment;
receiving a first traffic allotment for a first experiment variant of the first experiment and a second traffic allotment for a second experiment variant of the first experiment;
receiving a field value indicating that traffic to the first experiment is mutually excluded from other traffic to a second experiment; and
storing the first experiment identifier, the first traffic configuration setting, the first traffic allotment, the second traffic allotment in an experiment configuration file.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise:
performing a hashing function on data including a user identifier to obtain a hash value, the hash value used to determine whether to return a first web page corresponding to the first experiment variant or a second web page corresponding to the second experiment variant.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the data further includes the first experimental identifier.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the operations further comprise mapping the hash value to a bucket range to determine whether to return a web page corresponding to the first experiment variant or the second experiment variant.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the operations further comprise assigning a value to the user corresponding to a user identifier.

20. The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise receiving second data indicating that traffic to the second experiment is not mutually excluded from traffic to a third experiment.

Assignees

• Ebay Inc

Inventors

• Sahni Jasdeep Singh
• Madan Anil
• Nadig Deepak Seetharam
• Cheung Po
• Mistry Bhavesh
• Bodine John
• Lo Michael

Classifications

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3668 Primary

  Testing of software · CPC title

• G06F11/3696

  Methods or tools to render software testable · CPC title

• G06F16/954

  Navigation, e.g. using categorised browsing · CPC title

• G06F16/958

  Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking · CPC title