Generating a graphical user interface model from an image

Patent metadata

Field	Value
Publication number	US-10733754-B2
Application number	US-201715613122-A
Country	US
Kind code	B2
Filing date	Jun 2, 2017
Priority date	Jan 18, 2017
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

Techniques are disclosed for generating a GUI model for an application based upon requirements information for an application. The requirements information may include an image of a GUI screen (or multiple images of multiple screens) included in the GUI for the application. The GUI model generated based upon the input image may include information about the type and placement (layout) of GUI components in the GUI screen, and may also include information indicative of one or more functions to be assigned to one or more of the GUI components in the GUI screen. The GUI model may then be used to generate an implementation of the application GUI, including generating executable code that implements the GUI screen.

First claim

What is claimed is:

1. A method comprising:
receiving, at a computer system, an image of a graphical user interface (GUI) screen for an application;
generating, by the computer system, a GUI model for the GUI screen, the GUI model comprising information indicative of a plurality of GUI components included in the GUI screen, information indicative of a physical layout of the plurality of GUI components, and information indicative of a first function associated with a first GUI component from the plurality of GUI components;
wherein generating the GUI model comprises:
partitioning, by the computer system, the image into a plurality of partitions, the plurality of partitions including a first partition and a second partition, wherein the image is partitioned using an edge detection technique, wherein the edge detection technique detects edges that visually define areas of the image to partition the image into the plurality of partitions;
determining, by the computer system, a first set of rules for processing the first partition;
determining, by the computer system, a second set of rules for processing the second partition, wherein the second set of rules is different from the first set of rules;
processing, by the computer system, the first partition according to the first set of rules, the processing of the first partition comprising:
determining, by the computer system, based upon the first set of rules, a first set of two or more GUI components included in the first partition and their layout; and
determining, by the computer system, that the first function is to be associated with the first GUI component from the first set of two or more GUI components; and

processing, by the computer system, the second partition according to the second set of rules to determine a second set of two or more GUI components included in the second partition and their layout.

2. The method of claim 1 , wherein determining the first set of two or more GUI components included in the first partition and their layout comprises scanning pixels of the first partition to detect, within the first partition, one or more features that exceed a threshold probability of corresponding to a GUI component, and wherein the first function is determined based on the position of the first GUI component within the first partition.

3. The method of claim 2 , wherein scanning the pixels of a partition of the image comprises at least one of detecting one or more text regions within the partition using optical character recognition; and detecting one or more images within the partition using blob detection.

4. The method of claim 2 , wherein determining the first set of two or more GUI components included in the first partition and their layout further comprises, for a detected feature of the one or more detected features, identifying a known GUI component, from a set of known GUI components, that corresponds to the detected feature, wherein the identification is based on at least one of: one or more visual characteristics of the detected feature; and a position of the detected feature within the first partition.

5. The method of claim 2 , wherein determining the first and the second sets of two or more GUI components included in the first and the second partitions and their layouts comprises:
recursively segmenting each of the first and the second partitions to determine a respective hierarchy of one or more segments, wherein each link within the hierarchy of one or more segments denotes a containment relationship between segments connected by the link; and
scanning pixels of the second partition to detect, within the second partition, one or more features that exceed the threshold probability of corresponding to a GUI component,
wherein each of the first and second partitions includes at least one of a header, a footer, or a body of the GUI screen, and wherein a plurality of visual features of the GUI screen are erased prior to segmenting at least one of the first and the second partitions.

6. The method of claim 5 , wherein recursively segmenting the second partition to determine the hierarchy of one or more segments comprises:
initializing the hierarchy of one or more segments by placing the second partition at a root of the hierarchy of one or more segments;
detecting one or more top-level segments within the second partition by segmenting the second partition;
placing each of the one or more top-level segments as direct children of the root of the hierarchy of one or more segments; and
while the hierarchy of one or more segments has one or more unprocessed segments:
selecting an unprocessed segment from the hierarchy of one or more segments; attempt to detect sub-segments within the selected segment by segmenting the selected segment; and
when one or more sub-segments are detected within the selected segment, adding the one or more sub-segments to the hierarchy of one or more segments, wherein the one or more sub-segments are placed as direct children of the selected segment.

7. The method of claim 6 , wherein segmenting the selected segment comprises:
attempting to identify, from one or more detected edges that are located within the selected segment and one or more borders of the selected segment, one or more closed contours within the selected segment; and
for each of the one or more closed contours identified within the selected segment:
fitting an approximate polygon onto the closed contour; and
identifying a sub-segment that corresponds to the approximate polygon.

8. The method of claim 1 further comprising:
generating, by the computer system, using the GUI model, code that is executable by one or more processors and, when executed by the one or more processors causes the one or more processors to execute the first function when the first GUI component is interacted with.

9. The method of claim 1 , wherein the first function corresponds to one of:
a cancel function, wherein invoking the cancel function cancels an action that was invoked via the GUI;
a back function, wherein invoking the back function causes the GUI to return to a previous screen, mode, view, or window of the GUI;
an exit function, wherein invoking the exit function causes an underlying program associated with the GUI to close or halt execution on a device that is executing the underlying program;
a minimize function, wherein invoking the minimize function minimizes a screen or window of the GUI;
a maximize function, wherein invoking the maximize function maximizes the screen or window of the GUI;
an add function, wherein the add function is invoked for adding an item via the GUI;
a delete function, wherein the delete function is invoked for deleting a selected item via the GUI;
an option function, where invoking the option function launches a menu for configuring options or settings;
a drill-down function, where invoking the drill-down function causes the GUI to transition to a more specific screen, a more specific mode, a more specific view, or a more specific window of the GUI;
a save function, where invoking the save function causes the GUI to save a document;
an edit function, where invoking the edit function causes the GUI to facilitate editing of the document; and
a search function, where invoking the searching function causes the GUI to facilitate searching data.

10. The method of claim 1 , wherein the association between the first GUI component and the first function is determined from a dictionary that is specific to a domain of the first GUI component.

11. A non-transitory computer-readable storage memory storing a plurality of instructions executable by one or more processors, the plurality of instruct

Assignees

• Oracle Int Corp

Inventors

• Dayanandan Kailas
• John Philips George

Classifications

• G06F8/35 Primary

  model driven · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06F8/38

  for implementing user interfaces · CPC title