Data providers for annotations-based generic load generator

Patent metadata

Field	Value
Publication number	US-9870310-B1
Application number	US-201314133584-A
Country	US
Kind code	B1
Filing date	Dec 18, 2013
Priority date	Nov 11, 2013
Publication date	Jan 16, 2018
Grant date	Jan 16, 2018

Abstract

A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, data provider methods and data sources. The framework may generate and execute various test transactions and may call a data provider method to prepare data for the transaction and pass the prepared data to a transaction method. The framework may also load and parse test data from a data source and provide the test data to the data provider method for use when preparing data for the transaction.

First claim

What is claimed is:

1. A system, comprising:
one or more computers configured to implement a testing system, wherein the testing system comprises:
a transaction generation framework; and
a transaction creator module;
wherein the transaction creator module is dynamically bound to the transaction generation framework at runtime;
wherein the transaction generation framework is configured to discover, during runtime, source code annotations within the transaction creator module;
wherein the source code annotations comprise information identifying one or more transaction methods of the transaction creator module, wherein the information comprises a name of at least one of the one or more identified transaction methods, wherein individual ones of the one or more transaction methods cause a test transaction to be performed on a network-based production service configured to process requests from clients for the service via a network;
wherein the source code annotations further comprise information identifying one or more data provider methods of the transaction creator module, wherein individual ones of the one or more data provider methods provide data for one or more of the one or more transaction methods;
wherein the transaction generation framework is further configured to:
generate one or more test transactions to be performed on the service based on load step information; and
call, for at least one of the one or more test transactions, one of the data provider methods and one of the transaction methods for which the data provider methods provides data; and

wherein in response to the transaction generation framework calling one of the transaction methods, the transaction creator module is configured to communicate with the service to perform the respective test transaction.

2. The system of claim 1 , wherein the source code annotations further comprise information identifying a transaction type for individual ones of the one or more transaction methods and further comprise information associating at least one of the one or more data provider methods with one or more of the one or more transaction types.

3. The system of claim 2 , wherein said calling, for at least one of the one or more test transactions, one of the data provider methods, comprises calling a data provider method associated with a transaction type for the transaction method to be called for the test transaction.

4. The system of claim 1 , wherein the source code annotations further comprise information identifying a data source for one or more of the one or more data provider methods, wherein the data source comprises one or more records usable for one or more of the one or more test transactions.

5. A method, comprising:
performing, by one or more computers:
discovering, by a transaction generation framework at runtime, runtime-discoverable information within a transaction creator module, wherein the transaction generation framework is bound at runtime to the transaction creator module;
identifying based on the runtime-discoverable information, by the transaction generation framework, one or more transaction methods of the transaction creator module, wherein the runtime-discoverable information comprises a name of at least one of the one or more identified transaction methods, wherein at least one of the one or more transaction methods causes a test transaction to be performed on a network-based service under test, wherein the network-based production service is configured to process requests from clients for the service via a network; and
identifying based on the runtime-discoverable information, by the transaction generation framework, one or more data provider methods of the transaction creator module, wherein at least one of the one or more data provider methods provides test data for one or more of the one or more transaction methods.

6. The method of claim 5 , wherein the runtime-discoverable information comprises source code annotations identifying the one or more transaction methods and the one or more data provider methods.

7. The method of claim 5 , further comprising
generating, by the transaction generation framework, one or more test transactions to be performed on the service; and
for at least one of the one or more test transactions:
executing, by the transaction generation framework, one of the data provider methods and one of the transaction methods for which the data provider methods provides test data.

8. The method of claim 7 , further comprising:
preparing, by the transaction creator module in response to said executing one of the data provider methods, test data for the one of the transaction methods for which the data provider methods provides test data.

9. The method of claim 8 , further comprising:
identifying, by the transaction generation framework and based on the runtime-discoverable information, a data source comprising test data for the one or more transaction methods;
reading, by the transaction generation framework, a portion of the test data from the data source; and
wherein said executing one of the data provider methods comprises:
providing, by the transaction generation framework, the portion of the test data as an input parameter to the executed one of the data provider methods.

10. The method of claim 9 , further comprising:
receiving, by the transaction generation framework from the executed one of the data provider methods, prepared test data based on the test data provided as an input parameter; and
wherein said executing one of the transaction methods comprises:
providing, by the transaction generation framework, the prepared test data as an input parameter to the executed one of the transaction methods.

11. A non-transitory, computer-readable storage medium storing program instructions that when executed cause one or more computers to perform:
discovering, by a transaction generation framework at runtime, runtime-discoverable information within a transaction creator module, wherein the transaction generation framework is bound at runtime to the transaction creator module;
identifying, by the transaction generation framework based on the runtime-discoverable information, one or more transaction methods of the transaction creator module, wherein the runtime-discoverable information comprises a name of at least one of the one or more identified transaction methods, wherein individual ones of the one or more transaction methods cause a test transaction to be performed by a network-based service under test, wherein the network-based service is configured to process requests from clients for the service via a network;
identifying, by the transaction generation framework based on the runtime-discoverable information, one or more data provider methods of the transaction creator module, wherein at least one of the one or more data provider methods provides test data for one or more of the one or more transaction methods; and
identifying, by the transaction generation framework based on the runtime-discoverable information, a data source for one or more of the one or more data provider methods, wherein the data source comprises one or more records usable for one or more of the one or more test transactions.

12. The non-transitory, computer-readable storage medium of claim 11 , wherein the runtime-discoverable information further comprises information indicating field and/or record separating expressions for the data source.

13. The non-transitory, computer-readable storage medium of claim 11 , wherein the runtime-discoverable information further comprises information indicating whether the records within the data source

Assignees

• Amazon Tech Inc

Inventors

• Arguelles Carlos Alejandro
• Studd Kyle Thomas

Classifications

• G06F11/3414

  Workload generation, e.g. scripts, playback · CPC title

• G06F11/3452

  Performance evaluation by statistical analysis · CPC title

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• G06Q30/0206

  Price or cost determination based on market factors · CPC title

• G06F11/263

  Generation of test inputs, e.g. test vectors, patterns or sequences {; with adaptation of the tested hardware for testability with external testers} · CPC title