Time-limited dynamic testing pipelines

Patent metadata

Field	Value
Publication number	US-11119901-B2
Application number	US-201916675658-A
Country	US
Kind code	B2
Filing date	Nov 6, 2019
Priority date	Feb 1, 2019
Publication date	Sep 14, 2021
Grant date	Sep 14, 2021

Abstract

According to examples, an apparatus may include a processor that may generate automated and dynamic fail-fast testing pipelines that are efficiently executed to quickly identify tests for which changes to components such as an application will likely fail. The processor may train a classifier to predict whether changes to the application will fail a test procedure and use the classifier to generate machine-learned predictions of test outcomes to generate failure probabilities. The testing pipeline may be dynamically re-ordered based on the failure probabilities. The processor may also group the test procedures into lifecycle stages. Historical performance data may be used to identify time-limits by which to complete the test procedures of each lifecycle stage. Thus, the generated dynamic testing pipelines may be generated based on the likelihood of failures and test procedure duration.

First claim

What is claimed is:

1. An apparatus, comprising:
a processor; and
a non-transitory computer readable medium on which is stored instructions that when executed by the processor, cause the processor to:
receive an indication that a feature of a component has been altered, wherein the altered feature is to be tested based on a plurality of test procedures to be executed in a default test order;
predict that the component with the altered feature will fail a test procedure among the plurality of test procedures;
change the default test order to generate a dynamic test order that prioritizes the test procedure based on the prediction;
assign a first subset of the plurality of test procedures to a first lifecycle stage, the first subset including the test procedure;
order the first subset assigned to the first lifecycle stage according to an order in which the test procedures of the first subset appear in the dynamic test order;
identify a first time limit in which to perform the first subset assigned to the first lifecycle stage; and
generate a dynamic testing pipeline based on the plurality of test procedures, the dynamic test order, and the first time limit.

2. The apparatus of claim 1 , wherein a second subset of the plurality of test procedures is assigned to a second lifecycle stage, and wherein the instructions further cause the processor to:
determine that the first time limit has expired regardless of whether each test procedure of the first subset assigned to the first lifecycle stage has been completed; and
proceed to the second lifecycle stage based on the determination that the first time limit has expired.

3. The apparatus of claim 2 , wherein to proceed to the second lifecycle stage, the instructions further cause the processor to:
release the component with the altered feature for testing in the second lifecycle stage, wherein the component with the altered feature was previously prevented from testing in the first lifecycle stage due to expiration of the first time limit.

4. The apparatus of claim 2 , wherein to proceed to the second lifecycle stage, the instructions further cause the processor to:
generate an alert that indicates that the second lifecycle stage is to be initiated.

5. The apparatus of claim 2 , wherein the instructions further cause the processor to:
identify a second time limit in which the second subset assigned to the second lifecycle stage is to be performed; and
proceed to a third lifecycle stage when the second time limit has expired, regardless of whether each procedure within the second subset assigned to the second lifecycle stage has been completed.

6. The apparatus of claim 1 , wherein to identify the first time limit, the instructions further cause the processor to:
obtain historical information indicating a duration of each test procedure of the first subset; and
determine the first time limit based on the historical information of each test procedure of the first subset.

7. The apparatus of claim 6 , wherein to determine the first time limit based on the historical information of each test procedure of the first subset, the instructions further cause the processor to:
determine a probability of failure for each test procedure of the first subset; and
take into account within the first time limit the historical information of each test procedure of the first subset for which the probability of failure is greater than a threshold, such that the historical information of each test procedure of the first subset for which the probability of failure is not greater than the threshold is not taken into account within the first time limit.

8. The apparatus of claim 7 , wherein to determine the first time limit based on the historical information of each test procedure of the first subset, the instructions further cause the processor to:
determine an amount of time to complete each test procedure of the first subset; and
determine the first time limit based on the amount of time to complete each test procedure of the first subset.

9. The apparatus of claim 1 , wherein to predict that the component with the altered feature will fail the test procedure, the instructions further cause the processor to:
obtain information that describes the altered component; and
execute a machine-learned model of test outcomes based on the obtained information, wherein the machine-learned model generates the prediction.

10. The apparatus of claim 9 , wherein the instructions further cause the processor to:
obtain an outcome of the test procedure; and
re-train the machine-learned model based on the outcome.

11. A method comprising:
receiving, by a processor, an indication that a feature of a component has been altered, wherein the altered feature is to be tested based on a plurality of test procedures to be executed in a default test order;
generating, by the processor, a first probability that the component with the altered feature will fail a test procedure and a second probability that the component with the altered feature will fail another test procedure;
changing, by the processor, the default test order to generate a dynamic test order based on the first probability and the second probability;
assigning, by the processor, a first subset of the plurality of test procedures to a first lifecycle stage, the first subset including the test procedure;
assigning, by the processor, a second subset of the plurality of test procedures to a second lifecycle stage, the second subset including the other test procedure;
ordering, by the processor, the first subset of the first lifecycle stage according to an order in which the test procedures of the first subset appear in the dynamic test order;
ordering, by the processor, the second subset within the second lifecycle stage according to an order in which the test procedures of the second subset appear in the dynamic test order;
identifying, by the processor, a first time limit in which the first subset assigned to the first lifecycle stage is to be completed;
identifying, by the processor, a second time limit in which the second subset assigned to the second lifecycle stage is to be completed; and
generating, by the processor, a dynamic testing pipeline based on the plurality of test procedures, the dynamic test order, the first time limit, and the second time limit.

12. The method of claim 11 , further comprising:
training a machine-learned model of test outcomes based on previously executed dynamic testing pipelines, wherein the first probability and the second probability are each generated based on the machine-learned model.

13. The method of claim 11 , wherein generating the dynamic pipeline further comprises:
ordering the first lifecycle stage and the second lifecycle stage within the dynamic testing pipeline in a different lifecycle stage order than a default lifecycle stage order, according to an order in which a test procedure in each of the first lifecycle stage and the second lifecycle stage having a highest probability of failure appear in the dynamic test order.

14. The method of claim 13 , wherein the method further comprises:
permitting completion of as many of the first subset of the plurality of test procedures that are executable within the first time limit before proceeding to the second lifecycle stage, regardless of whether every test procedure of the first subset is completed within the first time limit; and
permitting completion of as many of the second subset of the plurality of test procedures that are executable within the second time limit before proceeding to a third lifecycle stage, regardless of

Assignees

• Micro Focus Llc

Inventors

• Jagannath Kishore
• Prabhu Namitha R

Classifications

• G06F11/3684 Primary

  for test design, e.g. generating new test cases · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title

• G06N5/04

  Inference or reasoning models · CPC title