Shared experiment control

Patent metadata

Field	Value
Publication number	US-10652770-B2
Application number	US-201815882942-A
Country	US
Kind code	B2
Filing date	Jan 29, 2018
Priority date	Jan 29, 2018
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

Systems and methods for providing a shared control group are disclosed. A request for a page is received and each user device is assigned to a shared control pool or one of a plurality of experiments. The shared control pool includes a first percentage of user devices and each of the experiments includes an experimental percentage. For each user device in the shared control pool, a base version of each element is transmitted; for each user device in a first experiment, a first variant element is transmitted; and for each user device assigned to a second experiment a second variant element is transmitted. A record indicative of interactions that occur after transmission is received and the interactions of a first subset of the shared control pool is compared with the devices assigned to the first experiment and a second subset is compared to the devices assigned to the second experiment.

First claim

What is claimed is:

1. A system, comprising:
a memory having instructions stored thereon, and a processor configured to read the instructions to:
receive a request for a page from a plurality of user devices;
assign each of the plurality of user devices to one of a shared control pool or one of a plurality of experiments for a first layer within the page, wherein the shared control pool includes a first percentage of a total number of the plurality of user devices and each of the plurality of experiments includes an experimental percentage of a total number of the plurality of user devices;
for each user device assigned to the shared control pool, transmit a base version of each element of the first layer to the user device;
for each user device assigned to a first experiment, transmit a first variant element associated with the first experiment and a base version for each additional element of the first layer to the user device;
for each user device assigned to a second experiment, transmit a second variant element associated with the second experiment and a base version for each additional element of the first layer to the user device;
receive, for each of the plurality of user devices, at least one record indicative of one or more interactions that occur after transmission of the base version or first variant element;
compare the one or more interactions of a first subset of user devices assigned to the shared control pool with the one or more interactions of each user device assigned to the first experiment; and
compare the one or more interactions of a second subset of user devices assigned to the shared control pool with the one or more interactions of each user device assigned to the second experiment.

2. The system of claim 1 , wherein the first subset of user devices assigned to the shared control group includes a first percentage of a total number of users assigned to the shared control group and the second subset of user devices assigned to the shared control group includes a second percentage of a total number of users assigned to the shared control group, wherein the first percentage and the second percentage are different.

3. The system of claim 1 , wherein at least one user device is included in each of the first subset of user devices assigned to the shared control group and the second subset of user devices assigned to the shared control group.

4. The system of claim 1 , wherein the page includes a second layer and each of the plurality of user devices is assigned to one of a shared control pool or one of a plurality of experiments for the second layer.

5. The system of claim 4 , wherein at least one user device is assigned to the shared control pool of the first layer and at least one experiment of the second layer.

6. The system of claim 1 , wherein the request for the page from each of the plurality of user devices includes an experimental variable, wherein the experimental variable is indicative of the assignment of a selected user device to the shared control pool or one of the plurality of experiments.

7. The system of claim 6 , wherein the processor is further configured to:
generate a cache key for transmittal to a cache server, wherein:
for each user device assigned to the first experiment, the cache key includes a variable indicative of the first variant element;
for each user devices assigned to the second experiment, the cache key includes a variable indicative of the second variant element; and
for each user device assigned to the shared control pool, the cache key does not include a variable for the first layer.

8. The system of claim 1 , wherein the first percentage of the total number of user devices assigned to the shared control group is equal to a largest of the experimental percentages.

9. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor cause a device to perform operations comprising:
receiving a request for a page from a plurality of user devices, wherein the page contains one or more elements in a first layer;
assigning each of the plurality of user devices to one of a shared control pool or one of a plurality of experiments for the first layer within the page, wherein the shared control pool includes a first percentage of the plurality of user devices and each of the plurality of experiments includes an experimental percentage of the plurality of user devices;
transmitting the one or more elements contained in the page to each of the plurality of user devices, wherein a base version of each of the one or more elements of the first layer is provided to each of the plurality of user devices assigned to the shared control pool, a first variant element associated with the first experiment and a base version for each of a remaining of the one or more elements of the first layer is provided to each of the plurality of user devices assigned to the first experiment, and a second variant element associated with the second experiment and a base version for each of the remaining of the one or more elements of the first layer is provided to each of the plurality of user devices assigned to the second experiment;
receiving, for each of the plurality of user devices, at least one record indicative of one or more interactions that occur after the transmission;
comparing the one or more interactions of a first subset of user devices assigned to the shared control pool with the one or more interactions of each user device assigned to the first experiment; and
comparing the one or more interactions of a second subset of user devices assigned to the shared control pool with the one or more interactions of each user device assigned to the second experiment.

10. The non-transitory computer readable medium of claim 9 , wherein the first subset of user devices assigned to the shared control group includes a first percentage of a total number of users assigned to the shared control group and the second subset of user devices assigned to the shared control group includes a second percentage of a total number of users assigned to the shared control group, wherein the first percentage and the second percentage are different.

11. The non-transitory computer readable medium of claim 9 , wherein at least one user device is included in each of the first subset of user devices assigned to the shared control group and the second subset of user devices assigned to the shared control group.

12. The non-transitory computer readable medium of claim 9 , wherein the page includes a second layer and each of the plurality of user devices is assigned to one of a shared control pool or one of a plurality of experiments for the second layer.

13. The non-transitory computer readable medium of claim 12 , wherein at least one user device is assigned to the shared control pool of the first layer and at least one experiment of the second layer.

14. The non-transitory computer readable medium of claim 9 , wherein the request for the page from each of the plurality of user devices includes an experimental variable, wherein the experimental variable is indicative of the assignment of a selected user device to the shared control pool or one of the plurality of experiments.

15. The non-transitory computer readable medium of claim 14 , wherein the instructions, when executed by the processor cause the device to perform further operations comprising:
generating a cache key for transmittal to a cache server, wherein:
for each user device assigned to the first experiment, the cache key includes a variable indicative of the first variant

Assignees

• Walmart Apollo Llc

Inventors

• Tang Anthony
• Bitondo Stephen
• Sharma Shriram
• Le Duy
• Subramanian Girish
• Kasireddi Swetharam

Classifications

• H04W28/16

  Central resource management; Negotiation of resources or communication parameters, e.g. negotiating bandwidth or QoS [Quality of Service] · CPC title

• H04W24/08 Primary

  Testing, {supervising or monitoring} using real traffic · CPC title

• H04W28/0215

  based on user or device properties, e.g. MTC-capable devices (services for machine-to-machine communication [M2M] or machine type communication [MTC] H04W4/70; wireless resource selection or allocation plan definition based on terminal or device properties H04W72/51) · CPC title

• H04W72/0493

  Electricity · mapped topic

• H04W72/53

  based on regulatory allocation policies · CPC title