Automatically correcting GUI automation using machine learning

Patent metadata

Field	Value
Publication number	US-10162741-B2
Application number	US-201715414133-A
Country	US
Kind code	B2
Filing date	Jan 24, 2017
Priority date	Jan 24, 2017
Publication date	Dec 25, 2018
Grant date	Dec 25, 2018

Abstract

A mechanism is provided in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement a user interface automation tool. The user interface automation tool executes a script to perform automation functions on user interface controls in a user interface of an application. Responsive to automation of a given user interface control failing, the user interface automation tool identifies a candidate user interface control that is the same as a user interface control expected in the script using a machine learning model. The user interface automation tool corrects the script to refer to the candidate user interface control to form a corrected script. The user interface automation tool performs a user interface function on the candidate user interface control according the corrected script.

First claim

What is claimed is:

1. A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement a user interface automation tool, the method comprising:
executing, by the user interface automation tool, a script to perform automation functions on user interface controls in a user interface of an application;
responsive to automation of a given user interface control failing, identifying, by the user interface automation tool, a candidate user interface control that is the same as a user interface control expected in the script using a machine learning model, wherein the machine learning model receives as input object properties of the given user interface control in the script, object properties of user interface objects in the user interface, code changes from a source control system, how many changes per day, number of defects, and a version of the application;
correcting, by the user interface automation tool, the script to refer to the candidate user interface control to form a corrected script; and
performing, by the user interface automation tool, a user interface function on the candidate user interface control according the corrected script.

2. The method of claim 1 , further comprising:
monitoring corrections to one or more user interface automation scripts;
recording the corrections to the one or more user interface automation scripts in a corrections data structure; and
training the machine learning model using the recorded corrections in the corrections data structure as training data.

3. The method of claim 2 , wherein the corrections data structure comprises for each of correction a previous set of properties for a changed user interface control and a current set of properties for the changed user interface control.

4. The method of claim 1 , wherein correcting the script comprises prompting a user to approve the candidate user interface object.

5. The method of claim 4 , wherein correcting the script further comprises correcting the script and recording the correction in a correction data structure in response to the user approving the candidate user interface object.

6. The method of claim 1 , wherein correcting the script comprises correcting object properties of the given user interface control.

7. The method of claim 1 , wherein correcting the script comprises correcting hidden tables associated with the given user interface control.

8. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a user interface automation tool, wherein the computer readable program causes the computing device to:
execute, by the user interface automation tool, a script to perform automation functions on user interface controls in a user interface of an application;
responsive to automation of a given user interface control failing, identify, by the user interface automation tool, a candidate user interface control that is the same as a user interface control expected In the script using a machine learning model, wherein, the machine learning model receives as input object properties of the given user interface control in the script, object properties of user interface objects in the user interface, code changes from a source control system, how many changes per day, number of defects, and a version of the application;
correct, by the user interface automation tool, the script to refer to the candidate user interface control to form a corrected script; and
perform, by the user interface automation tool, a user interface function on the candidate user interface control according the corrected script.

9. The computer program product of claim 8 , wherein the computer readable program further causes the computing device to:
monitor corrections to one or more user interface automation scripts;
record the corrections to the one or more user interface automation scripts in a corrections data structure; and
train the machine learning model using the recorded corrections in the corrections data structure as training data.

10. The computer program product of claim 9 , wherein the corrections data structure comprises for each of correction a previous set of properties for a changed user interface control and a current set of properties for the changed user interface control.

11. The computer program product of claim 8 , wherein correcting the script comprises prompting a user to approve the candidate user interface object.

12. The computer program product of claim 11 , wherein correcting the script further comprises correcting the script and recording the correction in a correction data structure in response to the user approving the candidate user interface object.

13. The computer program product of claim 8 , wherein correcting the script comprises correcting object properties of the given user interface control.

14. The computer program product of claim 8 , wherein correcting the script comprises correcting hidden tables associated with the given user interface control.

15. An apparatus comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a user interface automation tool, wherein the instructions cause the processor to:
execute, by the user interface automation tool, a script to perform automation functions on user interface controls in a user interface of an application;
responsive to automation of a given user interface control failing, identify, by the user interface automation tool, a candidate user interface control that is the same as a user interface control expected in the script using a machine learning model, wherein the machine learning model receives as input object properties of the given user interface control in the script, object properties of user interface objects in the user interface, code changes from a source control system, how many changes per day, number of defects, and a version of the application;
correct, by the user interface automation tool, the script to refer to the candidate user interface control to form a corrected script; and
perform, by the user interface automation tool, a user interface function on the candidate user interface control according the corrected script.

16. The apparatus of claim 15 , wherein the instructions further cause the processor to:
monitor corrections to one or more user interface automation scripts;
record the corrections to the one or more user interface automation scripts in a corrections data structure; and
train the machine learning model using the recorded corrections in the corrections data structure as training data.

17. The apparatus of claim 16 , wherein the corrections data structure comprises for each of correction a previous set of properties for a changed user interface control and a current set of properties for the changed user interface control.

18. The apparatus of claim 15 , wherein correcting the script comprises prompting a user to approve the candidate user interface object.

19. The apparatus of claim 18 , wherein correcting the script further comprises correcting the script and recording the correction in a correction data structure in response to the user approving

Assignees

• IBM

Inventors

• Kochura Nadiya
• Raghavan Vinodkumar
• Randall Jr Donald H
• Reedy Derek M
• Snow Timothy B

Classifications

• G06N20/00

  Machine learning · CPC title

• G06F11/3696 Primary

  Methods or tools to render software testable · CPC title

• G06F11/368

  for test version control, e.g. updating test cases to a new software version · CPC title

• G06F11/3664

  Physics · mapped topic

• G06N99/005

  Physics · mapped topic