System and method of handling complex experiments in a distributed system

Patent metadata

Field	Value
Publication number	US-10733078-B2
Application number	US-201514926007-A
Country	US
Kind code	B2
Filing date	Oct 29, 2015
Priority date	Oct 30, 2014
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

An experiment management system for an online visual design system including an experiment creator to receive an experiment specification with source material from an external source and to create an experiment definition based on inspection and analysis of the source material; an experiment integrator to integrate the experiment definition as an experiment into the experiment management system; an experiment configurer to configure a variant and population selection criteria for the experiment and an experiment manager to manage the experiment and at least one other experiment during the execution of the experiment at runtime of the online visual design system according to the variant and population selection.

First claim

What is claimed is:

1. A website building system (WBS) enabling web site designers to build and host websites for their end users, said system comprising:
a computing system having a memory and a processor; and
an experiment server running on said processor to manage multiple concurrent experiments running within said WBS, the experiment server comprising:
an experiment creator to receive an experiment specification to test features, components or system updates for said WBS and to create an experiment definition from said specification, said experiment creator to inspect and analyze source material and to modify compiled code and data objects of components of said website building system;
an experiment integrator to integrate said experiment definition as an experiment into said WBS wherein said experiment integrator also comprises an integration conflict resolver to resolve conflicts and dependency issues with features and components in already existing experiments within said WBS;
an experiment configurer to configure, according to a selection criteria, a target population of web site designers for said experiment, each of said criteria based at least on properties of said web site designers; and
an experiment manager to manage said experiment and at least one other experiment during the execution of said experiment at runtime of said WBS according to said target population wherein said experiment manager also comprises an active conflict resolver to detect and resolve conflicts and dependencies between features and components in said experiment and in said at least one other experiment running concurrently within said WBS.

2. The system according to claim 1 and also comprising an experiment analyzer to analyze results from said experiment during said runtime and to update said experiment manager accordingly.

3. The system according to claim 2 and also comprising:
an experiment terminator to terminate an experiment based on at least one of: said results from said experiment analyzer and manual override;
an experiment overrider to override said experiment selection based on interaction from a web site designer;
a data collector to collect results of said experiments for said experiment analyzer;
a baseline coordinator to update said experiment baseline based on analyzed results from said experiment analyzer; and
a variant modifier to recalculate and modify a variant selection during run time.

4. The system according to claim 3 and wherein said variant selection is based on at least one of: user ID, site ID and document ID.

5. The system according to claim 1 and also comprising a state map handler to integrate a state map with said experiment definition and wherein said state map defines how said experiment definition is run.

6. The system according to claim 5 and wherein said state map is at least one of: static, dynamic and modified.

7. The system according to claim 1 and wherein said experiment further includes modifications to data and WBS structures.

8. The system according to claim 1 and wherein said source material is at least one of: source code, binary code, web material and WBS data.

9. The system according to claim 8 and wherein said web material is at least one of: HTML, XML and CSS.

10. The system according to claim 1 and wherein said experiment is at least one of: an A/B test experiment, a multivariate experiment, a feature toggle experiment, a system upgrade and a marketing campaign.

11. The system according to claim 1 and wherein said source material comes from an external source, said source being at least one of: a developer, a third party application provider and a user.

12. The system according to claim 1 and wherein said experiment definition comprises at least one of an experiment header, an experiment script and experiment relationships definition.

13. The system according to claim 1 and wherein said active conflict resolver resolves conflicts between said experiment and at least one other experiment using experiment separation.

14. The system according to claim 1 and wherein said integration conflict resolver and said active conflict resolver resolve dependency circles.

15. The system according to claim 1 and wherein said experiment configurer comprises a population expander to expand said target population.

16. The system according to claim 1 and wherein said experiment manager comprises:
an experiment coordinator to coordinate the flow of experiments during runtime of said WBS;
an experiment selector to perform experiment selection for a given web site designer based on said target population criteria; and
an experiment implementer to implement said experiment selection.

17. The system according to claim 16 and wherein said experiment implementer implements at least one of: interpreted languages, compiled languages, data structures and bytecodes.

18. The system according to claim 1 wherein said criteria is also based on properties of said websites.

19. A method for a website building system (WBS) enabling web site designers to build and host websites for their end users; said method comprising:
managing multiple concurrent experiments running within said WBS, the managing comprising:
receiving an experiment specification to test features, components or system updates for said WBS and creating an experiment definition from said specification, said creating comprising inspecting and analyzing source material and modifying compiled code and data objects of components of said WBS;
integrating said experiment definition as an experiment into said WBS wherein said integrating also comprises integrally resolving conflicts and dependency issues with features and components in already existing experiments within said WBS;
configuring, according to a selection criteria, a target population of web site designers for said experiment each of said criteria based at least on properties of said web site designers; and
managing said experiment and at least one other experiment during the execution of said experiment at runtime of said WBS according to said target population;
wherein said managing comprises actively detecting and resolving conflicts and dependencies between features and components in said experiment and in said at least one of: said least one other experiment running concurrently within said WBS.

20. The method according to claim 19 and also comprising analyzing results from said experiment during said runtime and updating said managing accordingly.

21. The method according to claim 20 and also:
terminating an experiment based on at least one of: said results from said experiment analyzer and manual override;
overriding said experiment selection based on interaction from a web site designer;
collecting results of said experiments for said analyzing;
updating said experiment baseline based on analyzed results from said analyzing; and
recalculating and modifying a variant selection during run time.

22. The method according to claim 21 and wherein said variant selection is based on at least one of: user ID, site ID and document ID.

23. The method according to claim 19 and also comprising integrating a state map with said experiment definition and wherein said state map defines how said experiment definition is run.

24. The method according to claim 23 and wherein said state map is at least one of: static, dynamic and modified.

25. The method a

Assignees

• Wix Com Ltd

Inventors

• Mordo Aviran
• Gendler Talya
• Rozman Sagy
• Schwartz David
• Ben-David Ronen
• Nachman Omry

Classifications

• G06F11/3698 Primary

  Environments for analysis, debugging or testing of software · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06F8/34

  Graphical or visual programming · CPC title

• G06F11/3664 Primary

  Physics · mapped topic