Performance testing in a software deployment pipeline

Patent metadata

Field	Value
Publication number	US-10289539-B1
Application number	US-201314030715-A
Country	US
Kind code	B1
Filing date	Sep 18, 2013
Priority date	Sep 18, 2013
Publication date	May 14, 2019
Grant date	May 14, 2019

Abstract

Methods and systems for performance testing in a software deployment pipeline are disclosed. One or more performance tests are automatically performed on a build of a software product in a test environment in response to deploying the build to the test environment. One or more performance metrics are collected based on the performance tests. Based on the performance metrics, the build of the software product is accepted or rejected.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
deploying, based at least in part on a checkin of a software product, a build of the software product to a test environment prior to the build of the software product being deployed to a production environment;
generating a plurality of test transactions;
automatically performing a plurality of performance tests on the build of the software product in the test environment in response to said deploying the build of the software product to the test environment, wherein the performance tests include:
a performance test selected from a load test category, wherein each of the performance tests in the load test category comprises increasing a rate of transactions to the test environment to which the software product has been deployed until a predetermined latency threshold or an error rate threshold is reached, and wherein the performance tests are performed using at least one of the plurality of test transactions generated; and
another performance test selected from a sanity test category or a latency test category;

collecting a plurality of performance metrics based on the plurality of performance tests, wherein at least one performance test is configured in accordance with one or more respective pre-defined specifications;
determining whether the build of the software product passes or fails one or more of the performance tests based on one or more of the performance metrics; and
deploying the build of the software product to the production environment if the build of the software product passes the plurality of performance tests, and rejecting the build of the software product if the build of the software product fails one or more of the performance tests.

2. The method as recited in claim 1 , wherein performing the plurality of performance tests on the build of the software product comprises performing one or more sanity tests based on a typical amount of load, and wherein determining whether the build of the software product passes or fails one or more of the performance tests comprises determining whether an error rate for one or more transaction types exceeds a threshold.

3. The method as recited in claim 1 , wherein performing the performance tests on the build of the software product comprises performing one or more latency tests, and wherein determining whether the build of the software product passes or fails one or more of the performance tests comprises determining whether the plurality of performance metrics exceed a latency metric for one or more transaction types.

4. The method as recited in claim 1 , wherein performing the plurality of performance tests on the build of the software product comprises performing one or more load tests, and wherein determining whether the build of the software product passes or fails the performance tests comprises determining whether one or more hosts achieve a target transaction frequency or a target number of concurrent connections.

5. The method as recited in claim 1 , wherein determining whether the build of the software product passes or fails the performance tests based on the performance metrics comprises comparing one or more of the performance metrics to one or more performance metrics for a prior build of the software product.

6. The method as recited in claim 1 , further comprising:
performing one or more additional performance tests on a prior build of the software product, wherein the one or more additional performance tests are performed concurrently with the plurality of performance tests; and
collecting one or more additional performance metrics based on the one or more additional performance tests;
wherein determining whether the build of the software product passes or fails one or more of the performance tests based on one or more of the performance metrics comprises comparing at least one of the performance metrics with the one or more additional performance metrics.

7. A system, comprising:
one or more computers configured to implement a software build approval system, wherein the software build approval system comprises:
a transaction generator module configured to generate a plurality of test transactions;
a test environment deployment module configured to deploy, based at least in part on a checkin of a software product, a build of the software product to a test environment prior to the build of the software product being deployed to a production environment, wherein the test environment comprises one or more hosts configured to execute the build of the software product;
a performance testing module configured to:
apply a plurality of performance tests, using the plurality of test transactions generated by the transaction generator module, to the build of the software product in the test environment, wherein the performance tests include:
a performance test selected from a load test category, wherein for the performance tests in the load test category a rate of transactions to the one or more hosts configured to execute the build of the software product is increased until a predetermined latency threshold or an error rate threshold is reached; and
another performance test selected from a sanity test category or a latency test category;

collect a plurality of performance metrics based on the plurality of performance tests; and
determine whether to accept or reject the build of the software product based on the plurality of performance metrics; and

a production environment deployment module configured to:
deploy the build of the software product to a production environment in response to a determination to accept the build of the software product based on the plurality of performance metrics collected; and
reject the build of the software product in response to a determination to reject the build of the software product based on the plurality of performance metrics collected.

8. The system as recited in claim 7 , wherein the performance testing module is further configured to:
retrieve one or more performance metrics for a prior build of the software product; and
determine whether to accept or reject the build of the software product based on a comparison between at least one of the plurality of performance metrics for the build of the software product and the one or more performance metrics for the prior build of the software product.

9. The system as recited in claim 7 , wherein the performance testing module is further configured to:
apply one or more additional performance tests to a prior build of the software product in an additional test environment;
collect one or more additional performance metrics based on the additional performance tests; and
determine whether to accept or reject the build of the software product based on a comparison between the plurality of performance metrics and the one or more additional performance metrics.

10. The system of claim 7 , wherein at least one of the performance tests is configured to specify one or more test parameters to simulate a respective production load.

11. A system, comprising:
one or more computing devices having one or more hardware processors and memory and configured to implement one or more services, wherein the one or more services are configured to:
deploy, based at least in part on a checkin of a software product, a build of a software product to a test environment prior to the build of the software product being deployed to a production environment;
generate a plurality of transactions for the test environment;
automatically cause the build of the software product to process at least some of the generated transactions in performance tests of the test environment, where

Assignees

• Amazon Tech Inc

Inventors

• Arguelles Carlos Alejandro
• Carre Arthur
• Chandrasekharapuram Ramakrishnan Hariharan
• Tannous Ayham
• Wodicka Jason Scott

Classifications

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/076

  by exceeding a count or rate limit, e.g. word- or bit count limit · CPC title

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F8/60

  Software deployment · CPC title