Methods and systems of automatically associating text and control objects

Patent metadata

Field	Value
Publication number	US-12592094-B2
Application number	US-202117471633-A
Country	US
Kind code	B2
Filing date	Sep 10, 2021
Priority date	Sep 10, 2021
Publication date	Mar 31, 2026
Grant date	Mar 31, 2026

Abstract

Systems and methods of associating text with a graphical user interface (GUI) object are disclosed. Disclosed systems and methods include identifying a GUI object and a text string. A positional relationship between the GUI object and the text string is determined. Based on the positional relationship between the GUI object and the text string, a map with an indication associating the GUI object with the text string is updated.

First claim

What is claimed is:

1 . A computer-implemented method of associating text with a graphical user interface (GUI) object, the method comprising:
identifying, by one or more processors, a GUI object; identifying, by the one or more processors, a text string; determining, by the one or more processors, a positional relationship between the GUI object and the text string; based on the positional relationship between the GUI object and the text string, updating, by the one or more processors, a map with an indication associating the GUI object with the text string; and upon determining, by the one or more processors, that the text string and the GUI object are adjacent, building, by the one or more processors, a virtual envelope containing the text string and the GUI object, wherein the virtual envelope is sized such that at least one edge of the virtual envelope reaches an edge of the GUI object.

2 . The method of claim 1 , wherein identifying the GUI object comprises processing an image with a neural network.

3 . The method of claim 1 , further comprising determining, by the one or more processors, a confidence score indicating a degree of confidence as to whether the GUI object is associated with the text string.

4 . The method of claim 1 , wherein determining the positional relationship between the GUI object and the text string comprises:
determining, by the one or more processors, a distance between the text string and each of a plurality of GUI objects; and determining, by the one or more processors, a distance between the GUI object and each of a plurality of text strings.

5 . The method of claim 1 , further comprising determining, by the one or more processors, the virtual envelope does not overlap with one or more other GUI objects.

6 . The method of claim 1 , further comprising determining, by the one or more processors, the text string is one of above the GUI object and below the GUI object.

7 . The method of claim 1 , wherein the GUI object is identified within image data.

8 . A system for associating text with a graphical user interface (GUI) object, the system comprising:
one or more processors; and a computer-readable medium having encoded thereon computer-executable instructions configured to cause the one or more processors to:
identify a GUI object;
identify a text string;
determine a positional relationship between the GUI object and the text string;
based on the positional relationship between the GUI object and the text string, update a map with an indication associating the GUI object with the text string; and
determine, by the one or more processors, that the text string and the GUI object are adjacent, and build, by the one or more processors, a virtual envelope containing the text string and the GUI object, wherein the virtual envelope is sized such that at least one edge of the virtual envelope reaches an edge of the GUI object.

9 . The system of claim 8 , wherein identifying the GUI object comprises processing an image with a neural network.

10 . The system of claim 8 , wherein the instructions further cause the one or more processors to determine a confidence score indicating a degree of confidence as to whether the GUI object is associated with the text string.

11 . The system of claim 8 , wherein determining the positional relationship between the GUI object and the text string comprises:
determining a distance between the text string and each of a plurality of GUI objects; and determining a distance between the GUI object and each of a plurality of text strings.

12 . The system of claim 8 , wherein the instructions further cause the one or more processors to determine the virtual envelope does not overlap with one or more other GUI objects.

13 . The system of claim 8 , wherein the instructions further cause the one or more processors to determine the text string is one of above the GUI object and below the GUI object.

14 . The system of claim 8 , wherein the GUI object is identified within image data.

15 . A computer-readable storage device storing computer-executable instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform a method of associating text with a graphical user interface (GUI) object, the method comprising:
identifying a GUI object; identifying a text string; determining a positional relationship between the GUI object and the text string; based on the positional relationship between the GUI object and the text string, updating a map with an indication associating the GUI object with the text string; and upon determining that the text string and the GUI object are adjacent, building, by the one or more processors, a virtual envelope containing the text string and the GUI object, wherein the virtual envelope is sized such that at least one edge of the virtual envelope reaches an edge of the GUI object.

16 . The computer-readable storage device of claim 15 , wherein identifying the GUI object comprises processing an image with a neural network.

17 . The computer-readable storage device of claim 15 , further comprising determining a confidence score indicating a degree of confidence as to whether the GUI object is associated with the text string.

18 . The computer-readable storage device of claim 15 , wherein determining the positional relationship between the GUI object and the text string further comprises:
determining, by the one or more processors, a distance between the text string and each of a plurality of GUI objects; and determining, by the one or more processors, a distance between the GUI object and each of a plurality of text strings.

19 . The computer-readable storage device of claim 15 , further comprising determining the text string is one of above the GUI object and below the GUI object.

20 . The computer-readable storage device of claim 15 , wherein the GUI object is identified within image data.

Assignees

• Micro Focus Llc

Inventors

• Shang Er-Xin
• Fu Shuhui
• Guo Yi-Bin
• Lederer Niv

Classifications

• G06N3/02

  Neural networks · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F3/0481

  based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance · CPC title

• G06V10/82

  using neural networks · CPC title