Framework for UI automation based on graph recognition technology and related methods

Patent metadata

Field	Value
Publication number	US-11599449-B2
Application number	US-202117445512-A
Country	US
Kind code	B2
Filing date	Aug 20, 2021
Priority date	Jul 24, 2020
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

A GUI testing device may be configured to execute a testing state machine for interacting with a software application to generate an initial screen of a GUI. The GUI testing device may be configured to determine a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state. The current state may include an operation, and the operation may associate with a trigger target to operate on. The trigger may include a source state, a destination state, and a trigger target. The operation may include a user input operation, and an operation trigger target. The GUI testing device may be configured to perform the operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advance from the current state to a next state based upon the trigger.

First claim

The invention claimed is:

1. A computing system comprising:
a graphical user interface (GUI) testing device in communication with a computing device configured to execute a software application with an associated GUI, the GUI testing device configured to
execute a testing state machine for interacting with the software application to generate an initial screen of the GUI, the testing state machine comprising a plurality of states,
determine a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state, the current state comprising at least one operation, and at least one trigger associated with the at least one operation, the at least one trigger including a source state, a destination state, and a trigger target, the at least one operation comprising a user input operation, and an operation trigger target,
the determining of the current state comprising
applying a convolutional neural network (CNN) to generate a plurality of labels, and
finding the trigger including the matching trigger target in the plurality of labels, the matching trigger including a source state being the current state, and

perform the at least one operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advance from the current state to a next state based upon the at least one trigger.

2. The computing system of claim 1 wherein the GUI testing device is configured to determine a plurality of GUI elements in the initial screen; and wherein the matching trigger target comprises a matching target GUI element from the plurality of GUI elements.

3. The computing system of claim 1 wherein the GUI testing device is configured to determine the current state in the testing state machine by at least:
determining and applying a matching image template with the initial screen; and
finding the trigger including the matching trigger target in the matching image template, the matching trigger including a source state being the current state.

4. The computing system of claim 1 wherein the GUI testing device is configured to perform the at least one operation on the trigger target in a current screen to generate another screen of the GUI.

5. The computing system of claim 1 wherein the at least one operation comprises a plurality thereof; wherein the trigger target comprises a plurality thereof associated with the plurality of operations; and wherein the GUI testing device is configured to iteratively perform each operation on the trigger target to generate a plurality of next screens of the GUI.

6. The computing system of claim 1 wherein the user input operation comprises at least one of a keyboard input and a mouse input.

7. The computing system of claim 1 wherein the GUI testing device is configured to execute the testing state machine based upon a JavaScript Object Notation (JSON) file.

8. The computing system of claim 1 wherein the convolutional neural network comprises a pre-trained CNN.

9. The computing system of claim 1 wherein the GUI testing device is configured to train the CNN based upon a screenshot of the GUI, and an annotation file associated with the screenshot of the GUI.

10. The computing system of claim 1 wherein the determining of the current state comprises applying the CNN to generate a plurality of boxes associated with the plurality of labels, and a plurality of locations for the plurality of boxes.

11. A method for operating a graphical user interface (GUI) testing device in communication with a computing device configured to execute a software application with an associated GUI, the method comprising:
executing a testing state machine for interacting with the software application to generate an initial screen of the GUI, the testing state machine comprising a plurality of states;
determining a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state, the current state comprising at least one operation, and at least one trigger target associated with the at least one operation, the at least one trigger including a source state, a destination state, and a trigger target, the at least one operation comprising a user input operation, and an operation trigger target, the determining comprising
applying a convolutional neural network (CNN) to generate a plurality of labels, and
finding the trigger including the matching trigger target in the plurality of labels, the matching trigger including a source state being the current state; and

performing the at least one operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advancing from the current state to a next state based upon the at least one trigger.

12. The method of claim 11 further comprising determining a plurality of GUI elements in the initial screen; and wherein the matching trigger target comprises a matching target GUI element from the plurality of GUI elements.

13. The method of claim 11 wherein determining the current state in the testing state machine comprises: determining and applying a matching image template with the initial screen; and finding the trigger including the matching trigger target in the matching image template, the matching trigger target including a source state being the current state.

14. The method of claim 11 further comprising performing the at least one operation on the trigger target in a current screen to generate another screen of the GUI.

15. The method of claim 11 wherein the at least one operation comprises a plurality thereof; wherein the trigger target comprises a plurality thereof associated with the plurality of operations; and further comprising iteratively performing each operation on the trigger target to generate a plurality of next screens of the GUI.

16. The method of claim 11 wherein the user input operation comprises at least one of a keyboard input and a mouse input.

17. The method of claim 11 further comprising merging a plurality of text boxes in the initial screen of the GUI.

18. The method of claim 11 wherein the convolutional neural network comprises a pre-trained CNN.

19. The method of claim 11 further comprising training the CNN based upon a screenshot of the GUI, and an annotation file associated with the screenshot of the GUI.

20. The method of claim 11 wherein the determining comprises applying the CNN to a plurality of boxes associated with the plurality of labels, and a plurality of locations for the plurality of boxes.

Assignees

• Citrix Systems Inc

Inventors

• Lu Kai

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F11/3698 Primary

  Environments for analysis, debugging or testing of software · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3676

  for coverage analysis · CPC title