Computer-vision based execution of graphical user interface (GUI) application actions

Patent metadata

Field	Value
Publication number	US-10474440-B2
Application number	US-201816206625-A
Country	US
Kind code	B2
Filing date	Nov 30, 2018
Priority date	Sep 2, 2015
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

Using computer-vision based training information, a user interface (UI) component of an application-level user interface of an application and rendering coordinates of the UI component within the application-level user interface are recognized. A functional class that is mapped within the computer-vision based training information to the UI component and that is used to instantiate the UI component as part of the application-level user interface is identified in accordance with the computer-vision based training information. A replica object of the identified functional class is instantiated within a user interface container separately from the application. An operating system-level event that specifies a functional operation of the UI component and the recognized rendering coordinates of the UI component is generated from the instantiated replica object on an operating system event queue that provides inputs to the application.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
recognizing, using computer-vision based training information, a user interface (UI) component of an application-level user interface of an application and rendering coordinates of the UI component within the application-level user interface, where the computer-vision based training information is created by:
generating, by processing UI component training screenshot images, digitized image vectors as a training user interface data set;
applying machine learning to the training user interface data set;
training a graphical user interface (GUI) classifier module to recognize user interface images by use of the digitized image vectors;
mapping the trained digitized image vectors to user interface classes and user interface functional actions of the user interface classes; and
generating user interface execution scripts that specify the user interface classes and the user interface functional actions of replica object instances instantiated from the user interface classes; and

where the method further comprises:
identifying, in accordance with the computer-vision based training information, a functional class that is mapped within the computer-vision based training information to the UI component and that is used to instantiate the UI component as part of the application-level user interface;
instantiating a replica object of the identified functional class within a user interface container separately from the application; and
generating, from the instantiated replica object on an operating system event queue that provides inputs to the application, an operating system-level event that specifies a functional operation of the UI component and the recognized rendering coordinates of the UI component.

2. The computer-implemented method of claim 1 , where recognizing, using the computer-vision based training information, the user interface (UI) component of the application-level user interface of the application and the rendering coordinates of the UI component within the application-level user interface comprises:
capturing rendering data of the UI component;
classifying the captured rendering data of the UI component according to a user interface component type in accordance with the computer-vision based training information;
identifying, by the user interface component type, a class label and an instance identifier of the UI component mapped within the computer-vision based training information; and
identifying screen coordinates of the UI component within the rendering data of the UI component.

3. The computer-implemented method of claim 1 , where identifying, in accordance with the computer-vision based training information, the functional class that is mapped within the computer-vision based training information to the UI component and that is used to instantiate the UI component as part of the application-level user interface comprises:
identifying the functional class within the computer-vision based training information using an instance identifier of the UI component identified from captured rendering data of the UI component.

4. The computer-implemented method of claim 1 , where the application comprises a first application that is not in focus within a graphical user interface (GUI) to receive operating system inputs generated by an input device, where a second application is in focus within the GUI to receive operating system inputs generated by the input device, and where:
recognizing, using the computer-vision based training information, the UI component of the application-level user interface of the application and the rendering coordinates of the UI component within the application-level user interface comprises:
recognizing the UI component of the application-level user interface within one of a plurality of display buffers that is associated with a non-focused display view; and

generating, from the instantiated replica object on the operating system event queue that provides the inputs to the application, the operating system-level event that specifies the functional operation of the UI component and the recognized rendering coordinates of the UI component comprises:
generating the operating system-level event from the instantiated replica object as a non-focus-based input to the application without bringing the application-level user interface into focus within the GUI, where the second application retains the focus within the GUI to receive operating system inputs generated by the input device.

5. The computer-implemented method of claim 1 , further comprising creating the computer-vision based training information.

6. A system, comprising:
a memory; and
a processor programmed to:
recognize, using computer-vision based training information in the memory, a user interface (UI) component of an application-level user interface of an application and rendering coordinates of the UI component within the application-level user interface, where the computer-vision based training information is created by the processor being programmed to:
generate, by processing UI component training screenshot images, digitized image vectors as a training user interface data set;
apply machine learning to the training user interface data set;
train a graphical user interface (GUI) classifier module to recognize user interface images by use of the digitized image vectors;
map the trained digitized image vectors to user interface classes and user interface functional actions of the user interface classes; and
generate user interface execution scripts that specify the user interface classes and the user interface functional actions of replica object instances instantiated from the user interface classes; and

where the processor is further programmed to:
identify, in accordance with the computer-vision based training information, a functional class that is mapped within the computer-vision based training information to the UI component and that is used to instantiate the UI component as part of the application-level user interface;
instantiate a replica object of the identified functional class within a user interface container separately from the application; and
generate, from the instantiated replica object on an operating system event queue that provides inputs to the application, an operating system-level event that specifies a functional operation of the UI component and the recognized rendering coordinates of the UI component.

7. The system of claim 6 , where, in being programmed to recognize, using the computer-vision based training information in the memory, the user interface (UI) component of the application-level user interface of the application and the rendering coordinates of the UI component within the application-level user interface, the processor is programmed to:
capture rendering data of the UI component;
classify the captured rendering data of the UI component according to a user interface component type in accordance with the computer-vision based training information;
identify, by the user interface component type, a class label and an instance identifier of the UI component mapped within the computer-vision based training information; and
identify screen coordinates of the UI component within the rendering data of the UI component.

8. The system of claim 6 , where, in being programmed to identify, in accordance with the computer-vision based training information, the functional class that is mapped within the computer-vision based training information to the UI component and that is used to instantiate the UI component as part of the application-level user interface, the processor is programmed to:
identify the functional class with

Assignees

• IBM

Inventors

• Kochura Nadiya

Classifications

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F8/38 Primary

  for implementing user interfaces · CPC title

• G06F11/368

  for test version control, e.g. updating test cases to a new software version · CPC title