Time-limited dynamic testing pipelines

Patent metadata

Field	Value
Publication number	US-2020250078-A1
Application number	US-201916675658-A
Country	US
Kind code	A1
Filing date	Nov 6, 2019
Priority date	Feb 1, 2019
Publication date	Aug 6, 2020
Grant date	—

Abstract

According to examples, an apparatus may include a processor that may generate automated and dynamic fail-fast testing pipelines that are efficiently executed to quickly identify tests for which changes to components such as an application will likely fail. The processor may train a classifier to predict whether changes to the application will fail a test procedure and use the classifier to generate machine-learned predictions of test outcomes to generate failure probabilities. The testing pipeline may be dynamically re-ordered based on the failure probabilities. The processor may also group the test procedures into lifecycle stages. Historical performance data may be used to identify time-limits by which to complete the test procedures of each lifecycle stage. Thus, the generated dynamic testing pipelines may be generated based on the likelihood of failures and test procedure duration.

First claim

What is claimed is:

1 . An apparatus, comprising:
a processor; and a non-transitory computer readable medium on which is stored instructions that when executed by the processor, are to cause the processor to:
receive an indication that a feature of a component has been altered, wherein the altered feature is to be tested based on a plurality of test procedures to be executed in a default test order;
predict that the component with the altered feature will fail a test procedure among the plurality of test procedures;
change the default test order to generate a dynamic test order that prioritizes the test procedure based on the prediction;
identify a first time limit by which to perform the test procedure; and
generate a dynamic testing pipeline based on the plurality of test procedures, the dynamic test order, and the first time limit.

2 . The apparatus of claim 1 , wherein the instructions are further to cause the processor to:
assign a first subset of the plurality of test procedures to a first lifecycle stage, the first subset including the test procedure; and wherein to identify the first time limit, the instructions are to cause the processor to identify the first time limit for the first lifecycle stage such that the first subset is to be performed within the first time limit.

3 . The apparatus of claim 2 , wherein instructions are further to cause the processor to:
order the first subset within the first lifecycle stage based on the dynamic test order.

4 . The apparatus of claim 2 , wherein a second subset of the plurality of test procedures is assigned to a second lifecycle stage, and wherein the instructions are further to cause the processor to:
determine that the first time limit has expired; and proceed to the second lifecycle stage based on the determination that the first time limit has expired.

5 . The apparatus of claim 4 , wherein to proceed to the second lifecycle stage, the instructions are further to cause the processor to:
release the component with the altered feature for testing in the second lifecycle stage, wherein the component with the altered feature was previously prevented from testing in the second lifecycle stage prior to expiration of the first time limit.

6 . The apparatus of claim 4 , wherein to proceed to the second lifecycle stage, the instructions are further to cause the processor to:
generate an alert that indicates that the second lifecycle stage is to be initiated.

7 . The apparatus of claim 4 , wherein the instructions are further to cause the processor to:
identify a second time limit by which the second subset is to be performed; and proceed to a third lifecycle stage when the second time limit has expired.

8 . The apparatus of claim 2 , wherein to identify the first time limit, the instructions are further to cause the processor to:
obtain historical information indicating a duration of each test procedure of the first subset; and determine the first time limit based on the historical information.

9 . The apparatus of claim 1 , wherein to predict that the component with the altered feature will fail the test procedure, the instructions are further to cause the processor to:
obtain information that describes the altered component; and execute a machine-learned model of test outcomes based on the obtained information, wherein the machine-learned model generates the prediction.

10 . The apparatus of claim 9 , wherein the instructions are further to cause the processor to:
obtain an outcome of the test procedure; and re-train the machine-learned model based on the outcome.

11 . A method comprising:
receiving, by a processor, an indication that a feature of a component has been altered, wherein the altered feature is to be tested based on a plurality of test procedures to be executed in a default test order; generating, by the processor, a first probability that the component with the altered feature will fail a test procedure and a second probability that the component with the altered feature will fail another test procedure; changing, by the processor, the default test order to generate a dynamic test order based on the first probability and the second probability; identifying, by the processor, a first time limit by which to complete the test procedure; identifying, by the processor, a second time limit by which to complete the other test procedure; and generating, by the processor, a dynamic testing pipeline based on the plurality of test procedures, the dynamic test order, the first time limit, and the second time limit.

12 . The method of claim 11 , further comprising:
training a machine-learned model of test outcomes based on previously executed dynamic testing pipelines, wherein the first probability and the second probability are each generated based on the machine-learned model.

13 . The method of claim 11 , wherein generating the dynamic pipeline further comprises:
ordering the first lifecycle stage and the second lifecycle stage within the dynamic testing pipeline.

14 . The method of claim 13 , wherein the first lifecycle stage includes a first subset of the plurality of test procedures, the first subset including the test procedure, and wherein the second lifecycle stage includes a second subset of the plurality of test procedures, the method further comprising:
permitting completion of as many of the first subset of the plurality of test procedures that are executable within the first time limit before proceeding to the second lifecycle stage; and permitting completion of as many of the second subset of the plurality of test procedures that are executable within the second time limit before proceeding to a third lifecycle stage.

15 . A non-transitory computer readable medium on which is stored machine readable instructions that when executed by a processor, cause the processor to:
receive an indication that a feature of a component has been altered, wherein the altered feature is to be tested based on a plurality of test procedures to be executed in a default test order; determine a respective probability that the component with the altered feature will fail each of the plurality of test procedures; change the default test order to generate a dynamic test order based on the respective probabilities; determine, for each of the plurality of test procedures, historical information indicating a duration of each test procedure; identify a respective time limit by which to perform each of a plurality of lifecycle stages, each lifecycle stage including a corresponding subset of the plurality of test procedures and each respective time limit based on the historical information indicating the duration of each test procedure of the corresponding subset; and generate a dynamic testing pipeline based on the plurality of lifecycle stages, the dynamic test order, and the respective time limits.

16 . The non-transitory computer readable medium of claim 15 , wherein the plurality of test procedures are implemented as a development operations pipeline that provides automated testing of changes made to software.

17 . The non-transitory computer readable medium of claim 16 , wherein the plurality of lifecycle stages comprise one of: a preflight lifecycle stage, a development lifecycle stage, a testing lifecycle stage, a staging lifecycle stage, or a production stage,

18 . The non-transitory computer readable medium of claim 15 , wherein an order of the plurality of lifecycle stages is static, and wherein the instructions when

Assignees

• Micro Focus Llc

Inventors

• Jagannath Kishore
• Prabhu Namitha R

Classifications

• G06N20/00

  Machine learning · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3684 Primary

  for test design, e.g. generating new test cases · CPC title

• G06N5/04

  Inference or reasoning models · CPC title