Scalable testing in a production system with autoshutdown

Patent metadata

Field	Value
Publication number	US-8977903-B1
Application number	US-201213466955-A
Country	US
Kind code	B1
Filing date	May 8, 2012
Priority date	May 8, 2012
Publication date	Mar 10, 2015
Grant date	Mar 10, 2015

Abstract

A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

First claim

What is claimed is:

1. A system, comprising:
a network-based production service implemented by one or more computers, wherein the production service is configured to:
process requests from clients for the production service via a network;
capture production request data defining the requests;

a test system implemented by one or more computers, wherein the test system comprises:
one or more controllers configured to create test jobs according to a test plan for testing the production service, wherein the test plan specifies previously captured production request data to simulate a load on the production service, and wherein each job specifies a portion of the previously captured production request data;
a job queue configured to receive and queue test jobs from the one or more controllers according to the test plan;
one or more workers to access jobs from the job queue and replay production request data to the production service as specified by the jobs; and
an auto-shutdown module configured to monitor one or more system metrics and determine whether to shutdown the simulated load on the production service in response to the one or more system metrics.

2. The system as recited in claim 1 , wherein the one or more system metrics comprise the number of test jobs in the job queue, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number of test jobs in the job queue exceeding a given threshold.

3. The system as recited in claim 1 , wherein a test job is marked as untaken when added to the job queue;
wherein in response to a worker accessing a test job from the job queue, the test job is marked as taken in the job queue and the job remains in the job queue, wherein the job is ineligible to be given to another worker while marked as taken;
wherein in response to a worker successfully completing a test job, the test job is removed from the job queue; and
wherein in response to the test job not being successfully completed within a given time threshold, the test job is marked as untaken and becomes eligible to be given to another worker.

4. The system as recited in claim 3 , wherein the one or more system metrics comprise the number of untaken test jobs in the job queue, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number of untaken test jobs in the job queue exceeding a given threshold.

5. The system as recited in claim 3 , wherein the one or more system metrics comprise the number of taken test jobs in the job queue, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number of taken test jobs in the job queue exceeding a given threshold.

6. The system as recited in claim 1 , further comprising an auto-scaler configured to scale the number of said one or more workers to handle changes in the test load.

7. The system as recited in claim 6 , wherein the one or more system metrics comprise the current quantity of workers, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to an attempt to create an additional worker that would cause the current quantity of workers to exceed a given threshold.

8. The system as recited in claim 1 , wherein each of the one or more workers is configured to auto-shutdown if a number or percentage of unsuccessfully completed test jobs handled by the worker exceeds a given threshold.

9. The system as recited in claim 8 , wherein the one or more system metrics comprise a number or percentage of workers that auto-shutdown, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number or percentage of workers that auto-shutdown exceeding a given threshold.

10. The system as recited in claim 1 , wherein the one or more system metrics comprise one or more health metrics for the production service, wherein the one or more health metrics for the production service comprise memory usage, CPU usage, disk usage, or network usage.

11. The system as recited in claim 1 , further comprising a health monitoring service for the production service, wherein the auto-shutdown module is configured to call the health monitoring service to determine the one or more health metrics for the production service and automatically shutdown the simulated load on the production service in response to determining that the production service is overloaded according to the one or more health metrics.

12. A test system, comprising:
one or more computers configured to implement:
one or more controllers configured to create test jobs according to a test plan for testing a production service, wherein the test plan specifies previously captured production request data to simulate a load on the production service, and wherein each job specifies a portion of the previously captured production request data;
a job queue configured to receive and queue test jobs from the one or more controllers according to the test plan;
one or more workers to access jobs from the job queue and replay production request data to the production service as specified by the jobs; and
an auto-shutdown module configured to monitor one or more system metrics and determine whether to shutdown the simulated load on the production service in response to the one or more system metrics.

13. The test system as recited in claim 12 , wherein the one or more system metrics comprise the number of test jobs in the job queue, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number of test jobs in the job queue exceeding a given threshold.

14. The test system as recited in claim 12 , wherein a test job is marked as untaken when added to the job queue;
wherein in response to a worker accessing a test job from the job queue, the test job is marked as taken in the job queue and the job remains in the job queue, wherein the job is ineligible to be given to another worker while marked as taken;
wherein in response to a worker successfully completing a test job, the test job is removed from the job queue; and
wherein in response to the test job not being successfully completed within a given time threshold, the test job is marked as untaken and becomes eligible to be given to another worker.

15. The test system as recited in claim 14 , wherein the one or more system metrics comprise the number of untaken test jobs in the job queue, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number of untaken test jobs in the job queue exceeding a given threshold.

16. The test system as recited in claim 14 , wherein the one or more system metrics comprise the number of taken test jobs in the job queue, wherein the auto-shutdown module is configured to automatically shutdown the simulated load on the production service in response to the number of taken test jobs in the job queue exceeding a given threshold.

17. The test system as recited in claim 12 , further comprising an auto-scaler configured to scale the number of said one or more workers to handle changes in the test load.

18. The test system as recited in claim 12 , wherein the one or more system metrics comprise the current quantity of workers, wherein the auto-shutdown mo

Assignees

• Chandrasekharapuram Ramakrishnan H
• Arguelles Carlos A
• Amazon Tech Inc

Inventors

• Chandrasekharapuram Ramakrishnan H
• Arguelles Carlos A

Classifications

• G06F11/3668 Primary

  Testing of software · CPC title

• G06F11/3452

  Performance evaluation by statistical analysis · CPC title

• G06F2201/875

  Monitoring of systems including the internet · CPC title

• G06F11/3414

  Workload generation, e.g. scripts, playback · CPC title

• G06F11/263

  Generation of test inputs, e.g. test vectors, patterns or sequences {; with adaptation of the tested hardware for testability with external testers} · CPC title