Smart selection of test scripts for commodity testing on manufacturing floor

Patent metadata

Field	Value
Publication number	US-12013770-B2
Application number	US-201917426975-A
Country	US
Kind code	B2
Filing date	Feb 1, 2019
Priority date	Feb 1, 2019
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

In product testing, a script prioritization tool (102) is used to intelligently prioritize the execution sequence of test scripts. This tool creates a repository of test outputs from the executions of test scripts and analyzes the outputs to train and deploy a machine learning, ML, model that defines the priority of the scripts that may need to be executed and the scripts whose execution may be skipped without affecting the quality of testing. Scripts that are more likely to fail and/or are time consuming to execute are prioritized, while other scripts may be skipped. The ML model ranks the scripts based on the average execution time of the script, a count of the execution failures of the script, a count of the number of execution retries for the script, and the most recent failure time of the script. The scripts can be executed based on their rankings for efficiency and time-saving.

First claim

What is claimed is:

1. A method characterized by comprising:
determining ( 202 ), by a computing system ( 104 ), that a first unit ( 106 ) is to be tested through execution of a set of unit-specific test scripts;
identifying ( 203 ), by the computing system, one or more test scripts in the set having at least one of the following two attributes:
the identified test script is more likely to fail during execution, and
the identified test script are time consuming to execute;

testing ( 204 ), by the computing system, the first unit by executing the identified one or more test scripts while skipping the execution of the remaining test scripts in the set;
executing, by the computing system, each test script in the set a pre-determined number of times for every second unit until a pre-defined number of second units are tested, wherein each second unit is identical to the first unit;
collecting, by the computing system, an execution-specific result for each test script in the set that is executed the pre-determined number of times for every second unit, wherein the execution-specific result for a corresponding test script in the set includes the following:
a first data linking result of execution of the corresponding test script with a vendor of a respective second unit for which the corresponding test script is executed,
a second data linking result of execution of the corresponding test script with a supplier of the respective second unit for which the corresponding test script is executed, and
a third data linking result of execution of the corresponding test script with the computing system where the respective second unit is tested;

training, by the computing system, a machine learning, ML, model using all execution-specific results for each test script in the set that is executed the pre-determined number of times for every second unit; and
using, by the computing system, the trained ML model to identify the one or more test scripts to be executed for testing the first unit and those whose execution is to be skipped.

2. The method of claim 1 , wherein the first unit is one of the following:
a component of an electronic system; and
the electronic system.

3. The method of claim, wherein the determining comprises:
sensing, by the computing system, that the first unit is electrically connected to the computing system;
receiving ( 315 ), by the computing system, a part number associated with the first unit; and
based on the part number, retrieving ( 317 ), by the computing system, a list of the test scripts in the set of unit-specific test scripts.

4. The method of claim 1 , wherein the ML model is validated using a K-fold cross-validation method.

5. The method of claim 4 , wherein training the ML model comprises performing the following for each test script in the set that is executed the pre-determined number of times:
analyzing, by the computing system, all execution-specific results for the corresponding test script in the set to determine the following:
an average execution time of the corresponding test script, and
each failure time when execution of the corresponding test script fails; and

classifying, by the computing system, each failure time of the corresponding test script against the average execution time thereof into one of four quadrants, wherein each quadrant is defined by a corresponding pre-determined range of failure times along an x-axis thereof and a corresponding pre-determined range of average execution times along a y-axis thereof.

6. The method of claim 5 , wherein using the trained ML model comprises:
selecting, by the computing system, one or more quadrants associated with at least one of the following:
shorter failure times, and
higher average execution times;

for the selected one or more quadrants, ranking, by the computing system, each quadrant-specific test script based on a weighted evaluation of the following criteria:
the average execution time of the quadrant-specific test script,
a count of total number of failed executions of the quadrant-specific test script,
a count of total number of retried executions of the quadrant-specific test script for a frequently-failing Piece Part Identification, PPID, and
a most recent failure time of the quadrant-specific test script; and

based on an ascending order of the ranking, recommending ( 319 ), by the computing system, one or more quadrant-specific test scripts from the selected one or more quadrants as the identified one or more test scripts to be executed for testing the first unit.

7. The method of claim 6 , wherein the testing comprises:
executing, by the computing system, the identified one or more test scripts in the ascending order of ranking.

8. A computing system ( 104 ) characterized by comprising:
a memory ( 1104 , 1112 ) storing program instructions; and
a processing unit ( 1102 ) coupled to the memory and operable to execute the program instructions, which, when executed by the processing unit, cause the computing system to:
determine ( 202 ) that a first unit ( 106 ) is to be tested through execution of a set of unit-specific test scripts,
identify ( 203 ) one or more test scripts in the set having at least one of the following two attributes:
the identified test script is more likely to fail during execution, and
the identified test script are time consuming to execute,

test ( 204 ) the first unit by executing the identified one or more test scripts while skipping the execution of the remaining test scripts in the set,
execute each test script in the set a pre-determined number of times for every second unit until a pre-defined number of second units are tested, wherein each second unit is identical to the first unit;
collect an execution-specific result for each test script in the set that is executed the pre-determined number of times for every second unit, wherein the execution-specific result for a corresponding test script in the set includes the following:
a first data linking result of execution of the corresponding test script with a vendor of a respective second unit for which the corresponding test script is executed,
a second data linking result of execution of the corresponding test script with a supplier of the respective second unit for which the corresponding test script is executed, and
a third data linking result of execution of the corresponding test script with the computing system where the respective second unit is tested;

train a machine learning, ML, model using all execution-specific results for each test script in the set that is executed the pre-determined number of times for every second unit; and
use the trained ML model to identify the one or more test scripts to be executed for testing the first unit and those whose execution is to be skipped.

9. The computing system of claim 8 , wherein the computing system comprises a combination of the following:
a first computer ( 108 ) to which the first unit and every second unit are electrically connected, wherein the first computer is configured to test the first unit and every second unit by executing relevant test scripts; and
a second computer ( 104 ) operatively connected to the first computer, wherein the second computer is configured to train a machine learning, ML model and, based on the trained ML model, identify the one or more test scripts to be executed for testing the first unit.

10. The computing system of claim 9 , wherein the program instructions, upon execution by the processing unit, cause the computing system to perform the following for each test script in the set that is executed the pre-determined number of times:
analyze all execution-specific results for the corre

Assignees

• Dell Products Lp

Inventors

• Panikkar Shibi
• Anshul Shanir
• Dash Sudeshna
• Chin Tuck Meng
• Ji Dong

Classifications

• G06F11/277

  with comparison between actual response and known fault-free response · CPC title

• G06F11/2733

  Test interface between tester and unit under test · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F11/263 Primary

  Generation of test inputs, e.g. test vectors, patterns or sequences {; with adaptation of the tested hardware for testability with external testers} · CPC title