Systems and methods for measuring effectiveness of feature changes made within a store

Patent metadata

Field	Value
Publication number	US-11526824-B2
Application number	US-202015931052-A
Country	US
Kind code	B2
Filing date	May 13, 2020
Priority date	May 13, 2019
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

Systems and methods for determining whether a particular feature or change implemented in at least one test store causes a significant change as compared to one or more control stores are discussed. More particularly, techniques for using a time-series clustering algorithm to identify comparable sister stores to a store in which a feature change is being considered are described. Once the sister stores are identified a testing module can perform an A/B testing so as to validate whether a particular feature change being implemented in the test store causes a significant change as compared to the control stores.

First claim

We claim:

1. A system for determining effectiveness of feature changes made within a store, the system comprising:
a computer network coupled to a test store, and a plurality of stores;
at least one data storage device coupled to the computer network and holding a plurality of data sets, each data set associated with the plurality of stores, each store identified by a store identifier, and each store represented by time series data related to a specified variable, in a specified period of time;
a dashboard display coupled to the computer network and configured to receive as input a feature and one of the store identifiers to identify the test store in which to test the feature, the feature including at least one of: a change in product price, product location, product- related service and product availability, and wherein the dashboard display is further configured to display results of the test; and
a computing device coupled to the computer network and equipped with a memory and multiple processors configured to execute a testing module, the multiple processors forming a virtual machine to allow the test module to be executed on the multiple processors, the computing device communicatively coupled to the data storage device, and the dashboard display, wherein the testing module is further executed by the multiple processors for:
receiving, via the computer network, data associated with the store identifier for the test store and data associated with the plurality of stores;
transforming the time series data into respective feature vectors of the test store and the plurality of stores to be compared based on boundary conditions;
defining a Dynamic Time Warping (DTW) function according to at least a warping condition and the boundary conditions;
providing a computing kernel for the Dynamic Time Warping (DTW) function, in a programming language that rapidly performs alignment computations to allow cross-distance matrices to fit the memory, wherein performing the alignment computations comprises aligning two of a plurality of sequences of the respective feature vectors by iteratively warping a time axis, according to the warping condition, until an optimal match between the two sequences is found;
identifying, among the alignment computations, a best alignment between the two sequences by finding a path through a grid of the two sequences that minimizes a sum of distances between respective individual elements of the two sequences along the path;
identifying control stores from the plurality of stores for the test store identifier using the Dynamic Time Warping (DTW) function, and using the data associated with the test store identifier to rank a similarity of the control stores to the test store, wherein the control stores have comparable measurement values and follow a similar trend over a period of analysis as the test store based on a result of the Dynamic Time Warping (DTW);
receiving via the computer network at least one result from testing out the feature in the test store after the control stores are identified;
wherein the testing out of the feature is accomplished within the test store by at least one of: the changing of the product price of a selected product in the test store, the changing of the product location of the selected product within the test store, the changing of the product-related service at the test store, and the changing of the product availability of the selected product within the test store;
determining the at least one result of testing the feature in the test store is a change that is statistically significant from a related feature in the control stores; and
displaying, via the dashboard display, the at least one result of determining whether the change made in the test store is statistically significant;
wherein the feature is selectively implemented at one of the plurality of the stores based upon determining that the change made in the test store is statistically significant.

2. The system of claim 1 , wherein the testing module, when executed by the computing device, is further configured to generate scenarios and at least one comparison between scenario information.

3. The system of claim 2 wherein the dashboard display is configured to display the scenario information.

4. The system of claim 2 , wherein the scenarios are performed using historical orders made during the specified time period.

5. The system of claim 1 , wherein the dashboard display is further configured to display details regarding a lift of the test store and a distribution of the control stores according to the lift.

6. The system of claim 1 , further comprising a machine learning module executed by the computing device to:
segregate the control stores into similar clusters based on predefined factors;
segregate the control stores by a time series clustering algorithm to continuously evaluate each control store and then decide which cluster a control store falls into; and
change the predefined factors and re-segregate the control stores.

7. The system of claim 1 , wherein the testing module is executed by the computing device, to:
process the data associated with the plurality of stores using a time series clustering algorithm which segregates the plurality of stores into various clusters; and
use the processed data as criteria to generate the control stores for the test store.

8. The system of claim 1 wherein the at least one result of the feature for the test store is evaluated based on at least one of: sales data, footfalls and wait time.

9. A computer implemented method for determining effectiveness of feature changes made within a store, the method comprising:
storing, by at least one data storage device, a plurality of data sets, each data set associated with a plurality of stores, each store identified by a store identifier, and each store represented by time series data related to a specified variable, in a specified period of time;
receiving, at a dashboard display, input of a feature and one of the store identifiers to identify a test store in which to test the feature, the feature including at least one of: a change in product price, product location, product- related service and product availability, and wherein the dashboard display is further configured to display results of the test;
executing, by a computing device equipped with a memory and multiple processors, a testing module, to form, by the multiple processors, a virtual machine for allowing the test module to be executed on the multiple processors;
receiving, via a computer network, data associated with the store identifier for the test store and data associated with the plurality of stores;
transforming, by the multiple processors, the time series data into respective feature vectors of the test store and the plurality of stores to be compared based on boundary conditions;
defining, by the multiple processors, a Dynamic Time Warping (DTW) function according to at least a warping condition and the boundary conditions;
providing, by the multiple processors, a computing kernel for the Dynamic Time Warping (DTW) function, in a programming language that rapidly performs alignment computations to allow cross-distance matrices to fit the memory, wherein performing the alignment computations comprises aligning two of a plurality of sequences of the respective feature vectors by iteratively warping a time axis, according to the warping condition, until an optimal match between the two sequences is found;
identifying, among the alignment computations, a best alignment between the two sequences by finding a path through a grid of the two sequences that minimizes a sum of distances between respective individual elements of the two sequences alo

Assignees

• Walmart Apollo Llc

Inventors

• Kabra Paridhi
• Mishra Prateek
• Zhang Jingying
• Kapoor Nitin

Classifications

• G06Q10/06375 Primary

  Prediction of business process outcome or impact based on a proposed change · CPC title

• G06N20/00

  Machine learning · CPC title

• G06Q10/067

  Enterprise or organisation modelling · CPC title

• G06Q50/28

  Physics · mapped topic

• G06Q10/08 Primary

  Logistics, e.g. warehousing, loading or distribution; Inventory or stock management · CPC title