Identifying false positive automated tests

Patent metadata

Field	Value
Publication number	US-10331540-B2
Application number	US-201615085272-A
Country	US
Kind code	B2
Filing date	Mar 30, 2016
Priority date	Mar 30, 2016
Publication date	Jun 25, 2019
Grant date	Jun 25, 2019

Abstract

Technical solutions are described for identifying false positive tests. An example method includes receiving an input identifying a defect in a portion of a product code. The method also includes identifying a first automated test that is associated with the portion of the product code. The method also includes lowering a first test confidence value that is associated with the first automated test by a predetermined amount in response to the first automated test being a passing test. The method also includes marking the first automated test as a false positive test.

First claim

What is claimed is:

1. A computer implemented method for identifying false positive tests, the method implemented by a processor, and the method comprising:
receiving an input identifying a defect in a portion of a product code, the portion being one from a plurality of portions of the product code;
identifying, by the processor, a first automated test from a plurality of automated tests, the first automated test is associated with the portion of the product code based on a relatedness network that records a mapping between the automated tests and the portions of the product code, the mapping indicative that the first automated test and the portion of the product code were delivered to a source code management system as part of a common batch, maintaining the relatedness network comprising:
creating a test profile for each automated test from the plurality of automated tests;
determining that the portion of the product code and the first automated test are part of the common batch based on a difference between a first time stamp of the first automated test and a second timestamp of the portion of the product code being below a predetermined first threshold;
identifying a second automated test associated with the portion of the product code based on data from the source-code management system; and
mapping the portion of the product code and the second automated test with the first automated test using the test profile;
lowering, by the processor, a first test confidence value that is associated with the first automated test by a predetermined amount in response to the first automated test being a passing test; and
in response to the first test confidence value being below a predetermined second threshold:
marking, by the processor, the first automated test as a false positive test, wherein the marking indicates that the first automated test failed to identify the defect;
sending, by the processor, a notification that the first automated test is a false positive test; and
receiving an updated first automated test in response to sending the notification, wherein the updated first automated test is utilized in subsequent tests of the portion of the product code.

2. The computer implemented method of claim 1 , wherein, maintaining the relatedness network further comprises:
in response to a change to the portion of the product code, identifying the first automated test as being related to the portion of the product code based on metadata associated with the portion of the product code and the first automated test.

3. The computer implemented method of claim 1 , wherein the predetermined amount is a first predetermined amount, and the method further comprising:
lowering a second test confidence value that is associated with the second automated test by a second predetermined amount; and
in response to the second test confidence value being lower than the predetermined second threshold, marking the second automated test as a false positive test.

4. The computer implemented method of claim 3 , wherein identifying the second automated test is further based on a relatedness network that records a relationship between the first automated test and the second automated test.

5. The computer implemented method of claim 4 , further comprising, maintaining the relatedness network, wherein the maintaining comprises:
in response to receiving an updated first automated test, identifying the first automated test as being related to the second automated test based on metadata associated with the first automated test and the second automated test.

6. The computer implemented method of claim 3 , wherein the second automated test is identified as being related to the first automated test further based on the first automated test and the second automated test being associated to said portion of the product code.

7. The computer implemented method of claim 1 , further comprising:
executing the first automated test periodically;
in response to the first automated test passing for at least a predetermined number of times and the portion of the product code being changed, lowering the first test confidence value of the first automated test by the predetermined amount; and
in response to the first test confidence value being lower than the predetermined second threshold, marking the first automated test as a false positive test.

8. A system for identifying false positive tests, the system comprising:
a memory; and
a processor configured to:
receive an input identifying a defect in a portion of a product code, the portion being one from a plurality of portions of the product code;
identify a first automated test from a plurality of automated tests, the first automated test is associated with the portion of the product code based on a relatedness network that records a mapping between the automated tests and the portions of the product code, the mapping indicative that the first automated test and the portion of the product code were delivered to a source code management system as part of a common batch, maintaining the relatedness network comprising:
creating a test profile for each automated test from the plurality of automated tests;
determining that the portion of the product code and the first automated test are part of the common batch based on a difference between a first time stamp of the first automated test and a second time stamp of the portion of the product code being below a predetermined first threshold;
identifying a second automated test associated with the portion of the product code based on data from the source-code management system; and
mapping the portion of the product code and the second automated test with the first automated test using the test profile;
lower a first test confidence value that is associated with the first automated test by a predetermined amount in response to the first automated test being a passing test; and
in response to the first test confidence value being below a predetermined second threshold:
marking the first automated test as a false positive test, wherein the marking indicates that the first automated test failed to identify the defect;
sending, by the processor, a notification that the first automated test is a false positive test; and
receiving an updated first automated test in response to sending the notification, wherein the updated first automated test is utilized in subsequent tests of the portion of the product code.

9. The system of claim 8 , wherein the predetermined amount is a first predetermined amount, and the processor is further configured to:
lower a second test confidence value that is associated with the second automated test by a second predetermined amount; and
in response to the second test confidence value being lower than the predetermined second threshold, mark the second automated test as a false positive test.

10. The system of claim 9 , wherein the second automated test is identified as being related to the first automated test further based on the first automated test and the second automated test being associated to said portion of the product code.

11. The system of claim 8 , wherein the processor is further configured to:
execute the first automated test periodically;
lower the first test confidence value of the first automated test by the predetermined amount, in response to the first automated test passing for at least a predetermined number of times and the portion of the product code being changed; and
mark the first automated test as a false positive test, in response to the first test confidence value being lower than the predetermined second threshold.

12. A computer

Assignees

• IBM

Inventors

• Coulthard Adam P
• Leedham Neil W
• Woolley Mark A
• Wright Andy

Classifications

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3664

  Physics · mapped topic

• G06F11/368

  for test version control, e.g. updating test cases to a new software version · CPC title

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title