Dynamically adapting a test workload to accelerate the identification of performance issues

Patent metadata

Field	Value
Publication number	US-10176022-B2
Application number	US-201514976406-A
Country	US
Kind code	B2
Filing date	Dec 21, 2015
Priority date	Dec 21, 2015
Publication date	Jan 8, 2019
Grant date	Jan 8, 2019

Abstract

An improvement to the process for identifying software problems in performance testing is achieved by dynamically adjusting workloads in real-time to stress the functionality of an application suspected of causing a software problem.

First claim

What is claimed is:

1. A computer-implemented method for dynamically adapting a workload to expose performance issues, comprising:
using a processor device operably coupled with a memory, performing:
A) prior to a test run:
initializing a diagnosis policy comprising evaluation criteria used to assess performance of a system under test, the system comprising a plurality of applications with associated transactions; and
initializing an adjustment policy comprising rules for dynamically adjusting the workload for an identified transaction at run-time during the test run, wherein dynamically adjusting the workload comprises selectively re-tasking virtual users from a workload pool to stress functionality of the identified transaction; and

B) parallel to the test run, performing at pre-determined intervals during the test run:
collecting monitoring data for a pre-determined period of time;
applying the evaluation criteria from the diagnosis policy to the collected monitoring data to assess current system performance;
based upon identification of a current system performance issue after the applying, performing:
identifying a transaction associated with the current system performance issue;
matching the identified transaction associated with the current system performance issue with the rules associated with the identified transaction as set out in the adjustment policy; and
applying the rules from the adjustment policy to the identified transaction, thereby selectively adjusting the workload by increasing the workload in real-time for the identified transaction while decreasing the workload for other transactions not exhibiting a performance issue; and

storing the collected monitoring data.

2. The computer-implemented method of claim 1 wherein increasing the workload comprises increasing a number of virtual users tasked with running the identified transaction.

3. The computer-implemented method of claim 1 wherein initializing the diagnosis policy comprises specifying an outside evaluation tool to assess the performance.

4. The computer-implemented method of claim 1 wherein initializing the diagnosis policy comprises submitting application-dependent evaluation criteria targeted to measuring application performance metrics of a specific application under test.

5. The computer-implemented method of claim 1 wherein initializing the adjustment policy comprises providing: an initial workload, a maximum workload, and a delta workload indicating how many additional virtual users to add to stress functionality of the identified transaction, without exceeding the maximum workload.

6. The computer-implemented method of claim 5 wherein the delta workload specifies a percentage amount.

7. The computer-implemented method of claim 5 further comprising setting a period of time before an increased workload reverts to its initialized state specified as the initial workload in the adjustment policy.

8. The computer-implemented method of claim 1 wherein initializing the diagnosis policy comprises providing a set of application performance metrics and associated application performance thresholds, and identifying a test tool used to provide the set of application performance metrics.

9. The computer-implemented method of claim 8 wherein the set of application performance metrics comprise at least one of: throughput, response time, and cpu utilization rate.

10. An information processing system for adaptive workload adjustment of a system under test, the information processing system comprising:
a memory;
a processor device communicatively coupled to the memory, the processor device performing a method comprising:
A) prior to a test run:
initializing a diagnosis policy comprising evaluation criteria used to assess performance of the system under test, the system comprising a plurality of applications with associated transactions; and
initializing an adjustment policy comprising rules for dynamically adjusting a workload for an identified transaction at run-time during the test run, wherein dynamically adjusting the workload comprises selectively re-tasking virtual users from a workload pool to stress functionality of the identified transaction; and

B) parallel to the test run, performing at pre-determined intervals during the test run:
collecting monitoring data for a pre-determined period of time;
applying the evaluation criteria from the diagnosis policy to the collected monitoring data to assess current system performance;
based upon identification of a current system performance issue after the applying, performing:
identifying a transaction associated with the current system performance issue;
matching the transaction associated with the current system performance issue with the rules associated with the identified transaction as set out in the adjustment policy; and
applying the rules from the adjustment policy to the identified transaction, thereby selectively adjusting the workload by increasing the workload in real-time for the identified transaction, while decreasing the workload for other transactions not exhibiting a performance issue; and
storing the collected monitoring data.

11. The information processing system of claim 10 wherein the diagnosis policy specifies an outside evaluation tool to perform a diagnosis on the collected monitoring data.

12. The information processing system of claim 10 wherein initializing the adjustment policy comprises providing: an initial workload, a maximum workload, and a delta workload indicating how many additional virtual users to add to stress functionality of the identified transaction, without exceeding the maximum workload; and
wherein the method further comprises setting a period of time before an adjusted workload reverts to its initialized state specified as the initial workload in the adjustment policy.

13. The information processing system of claim 10 wherein initializing the diagnosis policy comprises submitting application-dependent evaluation criteria targeted to measuring application performance metrics of a specific application under test and the application performance metrics comprise at least one of: throughput, response time, and cpu utilization rate.

14. A computer program product for adaptive workload adjustment, the computer program product comprising:
a non-transitory computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit performing a method comprising:
A) prior to a test run:
initializing a diagnosis policy comprising evaluation criteria used to assess performance of a system under test, the system comprising a plurality of applications with associated transactions; and
initializing an adjustment policy comprising rules for dynamically adapting a test workload for an identified transaction at run-time during the test run, wherein dynamically adjusting the test workload comprises selectively re-tasking virtual users from a workload pool to stress functionality of the identified transaction; and

B) parallel to the test run, performing at pre-determined intervals during the test run:
collecting monitoring data for a pre-determined period of time;
applying the evaluation criteria from the diagnosis policy to the collected monitoring data to assess current system performance;
based upon identification of a current system performance issue after the applying, performing:
identifying a transaction associated with the current system performance issue;
matching the identified transaction associated with the current system performance issue with the rules associated with th

Assignees

• IBM
• Univ Dublin

Inventors

• Altman Erik R
• Assem Aly Salama Hitham Ahmed
• Mitchell Nicholas M
• O'Sullivan Patrick Joseph
• Portillo Dominguez Andres Omar
• Sweeney Peter F

Classifications

• G06F9/5083 Primary

  Techniques for rebalancing the load in a distributed system · CPC title

• G06F11/3672 Primary

  Test management · CPC title

• G06F11/3414

  Workload generation, e.g. scripts, playback · CPC title

• G06F11/3457

  Performance evaluation by simulation · CPC title

• G06F2201/865

  Monitoring of software · CPC title