System and method for testing in a production environment

Patent metadata

Field	Value
Publication number	US-9329915-B1
Application number	US-201213466943-A
Country	US
Kind code	B1
Filing date	May 8, 2012
Priority date	May 8, 2012
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

First claim

What is claimed is:

1. A system, comprising:
a network-based production service implemented by one or more computers, wherein the production service is configured to:
process requests from clients for the production service via a network; and
capture production request data defining the requests;

a data store configured to store the production request data captured at the production service; and
a test system implemented by one or more computers, wherein the test system comprises:
one or more controllers configured to create test jobs according to a test plan for testing the production service, wherein the test plan specifies production request data and a test profile for using the production request data to simulate a load on the production service, wherein the production request data was previously captured at the production service, and wherein each job specifies a portion of production request data;
a job queue configured to receive and queue test jobs from the one or more controllers;
wherein the one or more controllers are further configured to add the test jobs to the job queue according to the test plan; and
a plurality of workers configured to:
access jobs from the job queue;
access the production request data from the data store as specified in each job; and
replay the production request data to the production service.

2. The system as recited in claim 1 , wherein the production request data is stored encrypted in the data store.

3. The system as recited in claim 1 , wherein for each client request the production service is configured to tag the captured production request data with metadata describing an aspect of the production service to which the request pertains, and with timing information corresponding to the client request.

4. The system as recited in claim 1 , wherein the production request data is stored to the data store according to a standard timeframe, wherein the system further comprises a cleanup module configured to remove production request data from the data store that is older than a window of one or more standard timeframes such that a rolling window of production request data is maintained in the data store.

5. The system as recited in claim 1 , further comprising a worker scaling component configured to increase a number of the plurality of workers based on how many jobs are in the job queue or based on load metrics for the plurality of workers.

6. The system as recited in claim 5 , wherein the test system is configured to scale the number of the one or more controllers based on a size of the load to be simulated on the production service, wherein the number of controllers is scalable independently of the number of workers.

7. The system as recited in claim 1 , wherein the test profile specifies a rate profile at which the one or more controllers add jobs to the job queue.

8. A test system, comprising:
one or more computers configured to implement:
one or more controllers configured to create test jobs according to a test plan for testing a production service, wherein the test plan specifies production request data and a test profile for using the production request data to simulate a load on the production service, wherein the production request data was previously captured at the production service, and wherein each job specifies a portion of production request data;
a job queue configured to receive and queue test jobs from the one or more controllers;
wherein the one or more controllers are further configured to add the test jobs to the job queue according to the test plan; and
a load generator configured to implement one or more workers to access jobs from the job queue and replay production request data to the production service as specified by the jobs.

9. The test system as recited in claim 8 , wherein the production request data is stored in a data store and tagged in the data store with time information for a time period having a standard timeframe, wherein the controller is configured to create the test plan, wherein to create the test plan the controller is configured to specify production request data from one or more different time periods each having the standard timeframe.

10. The test system as recited in claim 9 , wherein to create the test plan the controller is further configured to specify production request data for a combination of a plurality of different time period each having the standard timeframe to create a test load for the standard timeframe larger than the production test data from any one of the plurality of different time periods.

11. The test system as recited in claim 9 , wherein to create the test plan the controller is further configured to specify the test profile, wherein the test profile indicates a rate at which the specified production request data is to be replayed to the production service.

12. The test system as recited in claim 8 , wherein the one or more controllers are configured to add the test jobs to the job queue at a rate specified by the test profile in the test plan.

13. The test system as recited in claim 8 , wherein each of the one or more workers is configured to execute test jobs from the job queue, wherein to execute one of the test jobs, the worker is configured to:
obtain production request data from a data store for a particular time period as specified in the test job;
transform time information in the production request data to a current time as specified in the test job; and
replay the transformed production request data to the production service to simulate new client requests.

14. The test system as recited in claim 8 , wherein in response to a worker accessing a test job from the job queue, the test job is marked as taken in the job queue and the job remains in the job queue, wherein the job is ineligible to be given to another worker while marked as taken.

15. The test system as recited in claim 14 , wherein in response to a worker successfully completing a test job, the test job is removed from the job queue; and
wherein in response to the job not being successfully completed within a given time threshold, the job is marked as untaken and becomes eligible to be given to another worker.

16. A method, comprising:
creating, by one or more controllers implemented on one or more computers, test jobs according to a test plan for testing a production service, wherein the test plan specifies production request data and a test profile for using the production request data to simulate a load on the production service, wherein the production request data was previously captured at the production service, and wherein each job specifies a portion of production request data;
adding, by the one or more controllers, the test jobs to a job queue according to the test plan; and
implementing, by a load generator implemented on one or more computers, one or more workers to access jobs from the job queue and replay production request data to the production service as specified by the jobs.

17. The method as recited in claim 16 , further comprising scaling the number of said one or more workers in response to one or more metrics, wherein the one or more metrics comprises a job quantity metric for how many jobs are in the job queue or a worker load metric of load on the plurality of workers.

18. The method as recited in claim 17 , further comprising implementing more workers in response to the job quantity metric indicating a number of test jobs in the job queue above a given threshold.

19. The method as recited in claim 17 , fu

Assignees

• Chandrasekharapuram Ramakrishnan H
• Arguelles Carlos A
• Amazon Tech Inc

Inventors

• Chandrasekharapuram Ramakrishnan H
• Arguelles Carlos A

Classifications

• G06F11/3495 Primary

  for systems · CPC title

• G06F11/3414

  Workload generation, e.g. scripts, playback · CPC title

• G06F11/00 Primary

  Error detection; Error correction; Monitoring (error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00) · CPC title

• G06F11/261

  by simulating additional hardware, e.g. fault simulation · CPC title