GUI component recognition apparatus, method and program

Patent metadata

Field	Value
Publication number	US-11954507-B2
Application number	US-202017908233-A
Country	US
Kind code	B2
Filing date	Mar 2, 2020
Priority date	Mar 2, 2020
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

A GUI component in a screen is recognized without analyzing a program code of application software.
A screen recognition device includes a GUI component form/position determination unit 11 that extracts a GUI component included in a display screen of application software, from the display screen, and that determines the form and position of the GUI component, and a GUI component kind identification unit 12 and a GUI component operation identification unit 13 that determine the kind of the GUI component and an operation that can be performed to the GUI component, based on a change in a mouse cursor and a change in the GUI component when the mouse cursor is superimposed on the GUI component.

First claim

The invention claimed is:

1. A screen recognition device comprising:
a processor; and
a storage medium having computer program instructions stored thereon, when executed by the processor, perform to:
perform a marking process on a graphical user interface (GUI) component in an active display screen provided by an application software, wherein the marking process distinguishes a background of the display screen from the GUI component in the display screen of the application software;
extract the GUI component included in the display screen of application software from the display screen;
superimpose a mouse cursor on the GUI component based on at least one of a form and a position of the GUI component extracted;
identify a change in the displayed symbol image of the mouse cursor;
determine that a kind of the extracted GUI component cannot be identified based only on the change in the displayed symbol image of the mouse cursor;
in response to the determining, determine the kind of the GUI component extracted and an operation that can be performed to the GUI component, based on the change in the mouse cursor and a change in the GUI component at the same time the mouse cursor is superimposed on the GUI component; and
moving the mouse cursor sequentially to each of the plurality of graphical user interfaces in the display screen based on the form and the position of each GUI component, wherein the mouse cursor is moved to and superimposed on each GUI component to determine the kind and the operation of the graphical user interface.

2. The screen recognition device according to claim 1 , wherein the computer program instructions further perform to sequentially superimpose the mouse cursor on the GUI component in the display screen of the application software, removes a range that is around the GUI component and in which a display form does not change, as a background, and extracts a remaining range of the GUI component.

3. The screen recognition device according to claim 1 , wherein the computer program instructions further perform to accept a preset combined key operation to the GUI component in the display screen of the application software, removes a range that is around the GUI component and in which a display form does not change, as a background, and extracts a remaining range of the GUI component.

4. A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the screen recognition device according to claim 1 .

5. A screen recognition method comprising:
a marking step of marking a graphical user interface (GUI) component in an active display screen provided by an application software, wherein the marking step distinguishes a background of the display screen from the GUI component in the display screen of the application software;
an extraction step of extracting the GUI component included in the display screen provided by the application software from the display screen;
a superimpose step of superimposing the mouse cursor over the GUI component in the display screen based on at least one of a form and a position of the GUI component;
an identification step of identifying a change in a displayed symbol image of the mouse cursor;
a determination step of determining that a kind of the GUI component cannot be identified based only on the change in the displayed symbol image of the mouse cursor, wherein in response to determining, determining the kind of the GUI component extracted in the extraction step and an operation that can be performed to the GUI component, based on the change in the displayed symbol image of the mouse cursor and a change in the GUI component at a same time the mouse cursor is superimposed on the GUI component; and
moving the mouse cursor sequentially to each of the plurality of GUI components in the display screen based on the form and the position of each GUI component, wherein the mouse cursor is moved to and superimposed on each GUI component to determine the kind and the operation of the GUI component.

6. The method of claim 5 further comprising:
identifying a range of the graphical user interface component in the display screen, wherein the range includes an area of the display screen having the graphical user interface component;
superimposing the mouse cursor on the range of the graphical user interface component identified in the display screen;
acquiring changes in a state of the range of the graphical user interface component and the mouse cursor at the time the mouse cursor is superimposed on the range of the graphical user interface;
creating a coordinate range based on the changes acquired in the state of the range of the graphical user interface component identified at the same time the mouse cursor is superimposed;
identifying the form of the range of the graphical user interface component; and
determining the position of the graphical user interface based on the coordinate range.

7. The method of claim 5 further comprises:
recognizing the displayed symbol image of the mouse cursor at the same time the mouse cursor is superimposed on the GUI component;
determining a setting of a property of the mouse cursor for a specific symbol image that correspond to a content of the GUI component at the same time the mouse cursor is superimposed on the graphical user interface component; and
identifying the content of the GUI component based on the displayed symbol image recognized for the mouse cursor.

8. The method of claim 5 , wherein the form of the GUI component includes one or more symbol images that corresponds to a content and/or operation of the GUI component.

9. A method for automatically recognizing a graphical user interface component in an active display screen, the method comprising:
having a display screen includes a plurality of graphical user interface components and a background provided by an application software, wherein the plurality of graphical user interface components incudes at least two graphical user interface components and the display screen is in active operation;
extracting a graphical user interface component of the plurality of graphical user interface components from the display screen provided by the application software;
identifying a first form and a position of the graphical user interface component extracted in the display screen;
superimposing a mouse cursor on the graphical user interface component in the display screen based on at least one of the first form and the position of the graphical user interface component in the display screen, wherein when the mouse cursor is superimposed on the graphical user interface component occurs at least one of the following occurs: a change in a display form of the mouse cursor and change in a display form of the graphical user interface component, wherein the display form of the mouse cursor includes a symbol image of the mouse cursor;
identifying a change in the display form of the mouse cursor;
determining that a kind of the graphical user interface component cannot be identified based on only the change in the display form of the mouse cursor;
in response to the determining, identifying the kind of the graphical user interface component along with an operation associated with the kind of the graphical user interface component in the display screen based on the change in the display form of the mouse cursor and the change in the display form the graphical user interface component; and
moving the mouse cursor sequentially to each of the plurality of graphical user interface components in the display screen based on the form and the position of each graphical user interface c

Assignees

• Nippon Telegraph & Telephone

Inventors

• Li Di
• Nozue Haruhisa
• Akashi Kazuaki
• Tayama Kenichi

Classifications

• G06F9/451 Primary

  Execution arrangements for user interfaces · CPC title

• G06F3/04812 Primary

  Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title