Identifying severity of test execution failures by analyzing test execution logs

Patent metadata

Field	Value
Publication number	US-2016283344-A1
Application number	US-201514671141-A
Country	US
Kind code	A1
Filing date	Mar 27, 2015
Priority date	Mar 27, 2015
Publication date	Sep 29, 2016
Grant date	—

Abstract

The method includes identifying a test report log for a regression test. The method further includes identifying one or more errors in the identified test report log. The method further includes determining a severity category for the one or more identified errors in the identified test report log. The method further includes determining a severity category for the identified test report log based on the determined severity category for the one or more identified errors in the identified test report log.

First claim

What is claimed is:

1 . A method for determining severity of execution test failures, the method comprising:
identifying, by one or more computer processors, a test report log for a regression test; identifying, by one or more computer processors, one or more errors in the identified test report log; determining, by one or more computer processors, a severity category for the one or more identified errors in the identified test report log; and determining, by one or more computer processors, a severity category for the identified test report log based on the determined severity category for the one or more identified errors in the identified test report log.

2 . The method of claim 1 , wherein, determining a severity category for the identified test report log comprises:
identifying, by one or more computer processors, a previous test report log with similar errors to the identified one or more errors in the identified test report log; and assigning, by one or more computer processors, a severity category assigned to the previous test report log to the identified test report log.

3 . The method of claim 1 , wherein determining a severity category for the identified test report log comprises:
identifying, by one or more computer processors, at least one preset error message that matches an error message included in the identified test report log and corresponds to a plurality of severity categories; and determining, by one or more computer processors, a severity category from plurality of severity categories that includes a most identical preset error message in relation to error messages in the identified test report log.

4 . The method of claim 1 , wherein determining a severity category for the identified test report log, further comprises:
determining, by one or more computer processors, a percentage completion that corresponds to the identified test report log based on the identified one or more errors; and assigning, by one or more computer processors, a severity category, from a plurality of severity categories, to the identified test report log based on the determined percentage completion.

5 . The method of claim 1 , wherein determining a severity category for the identified test report log, further comprises:
identifying, by one or more computer processors, a number of user interfaces without proper accessibility; and assigning, by one or more computer processors, a severity category, from a plurality of severity categories, to the identified test report log based on the determined identified number of user interfaces without proper accessibility.

6 . The method of claim 1 , wherein determining a severity category for the identified test report log, further comprises:
identifying, by one or more computer processors, one or more messages for trace and bug log files; and assigning, by one or more computer processors, a severity category, from a plurality of severity categories, to the identified test report log based on the identified one or more messages for trace and bug log files.

7 . The method of claim 1 , wherein determining a severity category for the identified test report log, further comprises:
determining, by one or more computer processors, a percentage completion that corresponds to the identified test report log based on the identified one or more errors; identifying, by one or more computer processors, a number of user interfaces without proper accessibility; and assigning, by one or more computer processors, a severity category, from a plurality of severity categories, to the identified test report log based on the determined percentage completion and the identified number of user interfaces without proper accessibility.

8 . A computer program product for determining severity of execution test failures, the computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising: program instructions to identify a test report log for a regression test; program instructions to identify one or more errors in the identified test report log; program instructions to determine a severity category for the one or more identified errors in the identified test report log; and program instructions to determine a severity category for the identified test report log based on the determined severity category for the one or more identified errors in the identified test report log.

9 . The computer program product of claim 8 , wherein, program instructions to determine a severity category for the identified test report log comprises program instructions to:
identify a previous test report log with similar errors to the identified one or more errors in the identified test report log; and assign a severity category assigned to the previous test report log to the identified test report log.

10 . The computer program product of claim 8 , wherein program instructions to determine a severity category for the identified test report log comprises program instructions to:
identify at least one preset error message that matches an error message included in the identified test report log and corresponds to a plurality of severity categories; and determine a severity category from plurality of severity categories that includes a most identical preset error message in relation to error messages in the identified test report log.

11 . The computer program product of claim 8 , wherein program instructions to determine a severity category for the identified test report log further comprises program instructions to:
determine a percentage completion that corresponds to the identified test report log based on the identified one or more errors; and assign a severity category, from a plurality of severity categories, to the identified test report log based on the determined percentage completion.

12 . The computer program product of claim 8 , wherein program instructions to determine a severity category for the identified test report log further comprises program instructions to:
identify a number of user interfaces without proper accessibility; and assign a severity category, from a plurality of severity categories, to the identified test report log based on the determined identified number of user interfaces without proper accessibility.

13 . The computer program product of claim 8 , wherein program instructions to determine a severity category for the identified test report log further comprises program instructions to:
identify one or more messages for trace and bug log files; and assign a severity category, from a plurality of severity categories, to the identified test report log based on the identified one or more messages for trace and bug log files.

14 . The computer program product of claim 8 , wherein program instructions to determine a severity category for the identified test report log further comprises program instructions to:
determine a percentage completion that corresponds to the identified test report log based on the identified one or more errors; identify a number of user interfaces without proper accessibility; and assign a severity category, from a plurality of severity categories, to the identified test report log based on the determined percentage completion and the identified number of user interfaces without proper accessibility.

15 . A computer system for determining severity of execution test failures, the computer system comprising:
one or more computer processors; one or more computer read

Assignees

• IBM

Inventors

• Brealey Christopher L
• Zacharias Shinoj

Classifications

• G06F11/0766

  Error or fault reporting or storing · CPC title

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/0709

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title

• G06F11/3409 Primary

  for performance assessment · CPC title

• G06F11/0781

  Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level · CPC title