Distributed software testing

Patent metadata

Field	Value
Publication number	US-9811451-B1
Application number	US-201514857252-A
Country	US
Kind code	B1
Filing date	Sep 17, 2015
Priority date	Sep 17, 2015
Publication date	Nov 7, 2017
Grant date	Nov 7, 2017

Abstract

Technologies are disclosed herein for distributed software testing. A software testing service is disclosed that can receive tests for software under test. The software testing service can determine a number of test instances to utilize to perform the tests. The determined number of test instances can then be provisioned, such as through the use of an on-demand computing service executing in a service provider network. The test instances can also be configured with the software under test and a test runner capable of performing the tests on the software. The software testing service can place the tests on a queue, such as a queue provided by a queue service. The test runners executing on the test instances can dequeue the tests and perform the tests on the software. Once the testing of the software under test has been completed, the test instances can be de-provisioned.

First claim

What is claimed is:

1. An apparatus, comprising:
a processor; and
a non-transitory computer-readable storage medium having instructions stored thereupon which are executable by the processor and which, when executed, cause the apparatus to:
receive a request to perform a plurality of tests on software, the request comprising an indication of testing preferences;
determine, based at least in part on the testing preferences, a number of test instances to utilize for performing the plurality of tests on the software;
responsive to determining the number of the test instances, automatically cause the number of the test instances to be provisioned by an on-demand computing service executing in a service provider network, wherein the test instances include one or more of a virtual machine, or a container;
cause a test runner to be deployed to the test instances;
cause the software to be deployed to the test instances;
place the plurality of tests on a queue provided by a queue service executing in the service provider network, wherein the test runner dequeues the plurality of tests from the queue and performs the plurality of tests on the software in parallel on the test instances;
determine that the plurality of tests have completed; and
cause the test instances to be de-provisioned by the on-demand computing service in response to determining that the plurality of tests have completed.

2. The apparatus of claim 1 , wherein the testing preferences specify the number of test instances to utilize for performing the plurality of tests on the software.

3. The apparatus of claim 1 , wherein the testing preferences specify a maximum execution time for the plurality of tests, and wherein the number of test instances to utilize for performing the plurality of tests on the software is determined based, at least in part, on historical test execution time data for the plurality of tests.

4. The apparatus of claim 1 , wherein the testing preferences specify a maximum execution time for the plurality of tests, and wherein the number of test instances to utilize for performing the plurality of tests on the software is determined based, at least in part, on historical test execution time data for tests similar to the plurality of tests.

5. The apparatus of claim 1 , wherein the non-transitory computer-readable storage medium has further instructions stored thereupon to:
compute an estimated amount of time for completion of the plurality of tests;
determine that the estimated amount time for completion of the plurality of tests is less than a desired amount of time for completion of the plurality of tests; and
cause one or more of the test instances to be de-provisioned by the on-demand computing service in response to determining that the estimated amount of time for completion of the plurality of tests is less than the desired amount of time for completion of the plurality of tests.

6. The apparatus of claim 1 , wherein the non-transitory computer-readable storage medium has further instructions stored thereupon to:
compute an estimated amount of time for completion of the plurality of tests;
determine that the estimated amount time for completion of the plurality of tests is greater than a desired amount of time for completion of the plurality of tests; and
cause one or more additional test instances to be provisioned by the on-demand computing service in response to determining that the estimated amount of time for completion of the plurality of tests is greater than the desired amount of time for completion of the plurality of tests.

7. A computer-implemented method for performing a plurality of tests on software, the method comprising:
receiving a request to perform a plurality of tests on software;
determining a number of test instances to utilize for performing the plurality of tests on the software;
responsive to determining the number of the test instances, automatically causing the number of test instances to be provisioned, wherein the test instances include one or more of a virtual machine, or a container;
causing a test runner to be deployed to the test instances;
causing the software to be provisioned to the test instances;
distributing the plurality of tests to the test runner deployed to the test instances wherein the test runner performs the plurality of tests on the software in parallel on the test instances;
determining that the plurality of tests have completed; and
causing the test instances to be de-provisioned in response to determining that the plurality of tests have completed.

8. The computer-implemented method of claim 7 , wherein the request to perform the plurality of tests comprises an indication of testing preferences, and wherein the number of test instances to utilize for performing the plurality of tests on the software is determined based, at least in part, on the testing preferences.

9. The computer-implemented method of claim 8 , wherein the testing preferences specify the number of test instances to utilize for performing the plurality of tests on the software.

10. The computer-implemented method of claim 8 , wherein the testing preferences specify a maximum execution time for the plurality of tests, and wherein the number of test instances to utilize for performing the plurality of tests on the software is determined based, at least in part, on historical test execution time data for the plurality of tests.

11. The computer-implemented method of claim 8 , wherein the testing preferences specify a maximum execution time for the plurality of tests, and wherein the number of test instances to utilize for performing the plurality of tests on the software is determined based, at least in part, on historical test execution time data for tests similar to the plurality of tests.

12. The computer-implemented method of claim 7 , wherein causing the determined number of test instances to be provisioned comprises instructing an on-demand computing service executing in a service provider network to provision the determined number of test instances, and wherein causing the test instances to be de-provisioned comprises instructing the on-demand computing service to de-provision the test instances.

13. The computer-implemented method of claim 7 , wherein distributing the plurality of tests available to the test runner deployed to the test instances comprises placing the tests on a queue provided by a queue service executing in a service provider network, and wherein the test runner is configured to dequeue a test from the queue and execute the dequeued test on the software.

14. The computer-implemented method of claim 7 , further comprising:
computing an estimated amount of time for completion of the plurality of tests;
determining that the estimated amount time for completion of the plurality of tests is less than a desired amount of time for completion of the plurality of tests; and
causing one or more of the test instances to be de-provisioned in response to determining that the estimated amount of time for completion of the plurality of tests is less than the desired amount of time for completion of the plurality of tests.

15. The computer-implemented method of claim 7 , further comprising:
computing an estimated amount of time for completion of the plurality of tests;
determining that the estimated amount time for completion of the plurality of tests is greater than a desired amount of time for completion of the plurality of tests; and
causing one or more additional test instances to be provisioned in response to determining that the estimated amount of time for completion of

Assignees

• Amazon Tech Inc

Inventors

• Arguelles Carlos Alejandro
• Griffin David Robert

Classifications

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title