Automated code generation

Patent metadata

Field	Value
Publication number	US-10489126-B2
Application number	US-201815996311-A
Country	US
Kind code	B2
Filing date	Jun 1, 2018
Priority date	Feb 12, 2018
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

Techniques are disclosed for automating GUI development from a GUI screen image that includes text information and one or more graphic user interface components. The GUI screen image is analyzed to extract text information and to identify the UI components included in the GUI screen. One or more text regions in the GUI screen image are detected and are replaced with placeholders. Images of one or more graphic user interface components in the GUI screen are extracted from the GUI screen image and are classified using a machine learning-based classifier. A GUI model is generated for the GUI based upon the classification results, locations of the one or more text regions, and locations of the one or more graphic user interface components. The generated model can then be used to generate one or more implementations (e.g., executable code) of the GUI, possibly for various platforms in different programming languages.

First claim

What is claimed is:

1. A method comprising:
detecting, by a computer system, from an input image, a graphic user interface (GUI) screen image depicting a GUI screen of a GUI;
detecting, by the computer system, a first region of the GUI screen image that includes a first text content item;
determining, by the computer system, a location of the first region within the GUI screen image;
replacing, by the computer system, the first text content item in the first region with a placeholder that does not include text content;
detecting, by the computer system, a first user interface (UI) component located in a second region of the GUI screen image, wherein the second region includes the first region with the placeholder that replaces the first text content item;
determining, by the computer system, a location of the first UI component within the GUI screen image;
determining, by the computer system using a machine learning-based classifier, a UI component type for the first UI component, wherein the machine learning-based classifier is trained using training data comprising a plurality of training images, each training image in the plurality of training images comprising a UI component, the training data further comprising, for each training image, information identifying a UI component type for the UI component in the training image;
generating, by the computer system, a GUI model that is usable for generating code for implementing the GUI, the GUI model comprising information for the first UI component and information for the first text content item, wherein the information for the first UI component includes information indicative of the UI component type determined for the first UI component and the location of the first UI component within the GUI screen image; and
creating one or more implementations of the GUI model, the one or more implementations executable by one or more processors to implement the GUI.

2. The method of claim 1 , wherein:
detecting the first UI component comprises detecting, by the machine learning-based classifier, the first UI component in the GUI screen image; and
the training data further comprises, for each training image, a location of a UI component within the training image.

3. The method of claim 1 , further comprising:
based upon the location of the first region within the GUI screen image and the location of the first UI component within the GUI screen image, grouping the first text content item with the first UI component; and
wherein generating the GUI model further comprises including, in the GUI model, information indicative of the grouping.

4. The method of claim 1 , further comprising:
detecting, by the computer system, a third region of the GUI screen image that includes a second text content item;
determining, by the computer system, a location of the third region within the GUI screen image;
detecting, by the computer system, a second UI component located in the GUI screen image;
determining a location of the second UI component within the GUI screen image;
determining, using the machine learning-based classifier, a UI component type for the second UI component;
grouping the first text content item, the second text content item, the first UI component, and the second UI component based upon the location of the first region, the location of the third region, the UI component type and location of the first UI component, and the UI component type and location of the second UI component; and
determining a layout of the GUI screen based upon the grouping,
wherein generating the GUI model further comprises including, in the GUI model, information indicative of the grouping and the layout of the GUI screen.

5. The method of claim 4 , further comprising:
determining, based upon the location of the third region within the GUI screen image, that the second text content item is not associated with any UI component in the GUI screen image; and
determining that the second text content item is indicative of an action,
wherein generating the GUI model comprises indicating, in the GUI model, that the second text content item within the third region of the GUI screen image is clickable text, wherein clicking of the second text content item initiates the action.

6. The method of claim 1 , wherein the first region comprises the first text content item on a background, the method further comprising:
generating a histogram based on intensity values of pixels in the first region of the GUI screen image;
determining, from the histogram, that intensity values of pixels for the first text content item are higher than intensity values of pixels of the background based upon a number of pixels corresponding to the first text content item and a number of pixels corresponding to the background in the first region of the GUI screen image;
inverting the intensity values of the pixels in the first region, wherein the inverting causes the intensity values of the pixels for the first text content item to be lower than the intensity values of the pixels of the background in the first region; and
recognizing the first text content item in the first region after performing the inverting.

7. The method of claim 6 , further comprising:
converting, before generating the histogram, the first region of the GUI screen image from an RGB sub-image to a binary sub-image based upon an intensity value of each pixel in the first region.

8. The method of claim 1 , wherein:
the placeholder includes a pre-defined pattern of pixels; and
determining the UI component type for the first UI component comprises classifying the second region of the GUI screen image based upon presence of the placeholder in the second region and the pre-defined pattern of pixels in the placeholder.

9. The method of claim 1 , wherein the placeholder is a pre-defined pattern of pixels enabling boundaries of the location of the first text content item in the first region to be determined from the GUI screen image.

10. The method of claim 9 , wherein pixel values of the pre-defined pattern of pixels include a pre-defined value or a pixel value of a pixel in a background of the first region.

11. The method of claim 1 , wherein generating the one or more implementations of the GUI based upon the GUI model comprises:
generating, using the GUI model, a first implementation of the GUI for a first platform; and
generating, using the GUI model, a second implementation of the GUI for a second platform, wherein the second platform is different from the first platform.

12. The method of claim 1 , wherein generating the one or more implementations of the GUI based upon the GUI model comprises:
generating, using the GUI model, a first implementation of the GUI in a first programming language; and
generating, using the GUI model, a second implementation of the GUI in a second programming language, wherein the second programming language is different from the first programming language.

13. The method of claim 1 , wherein generating the one or more implementations of the GUI based upon the GUI model comprises:
generating the one or more implementations of the GUI using the GUI model and one or more code generation templates, each code generation template associated with a platform or a programming language.

14. The method of claim 1 , wherein generating the GUI model comprises storing information of the GUI model in a JavaScript Object Notation (JSON) format.

15. The method of claim 1 , wherein the machine learning-based classifier includes an image histogram-based nonlinear support vector machine classifier or an artif

Assignees

• Oracle Int Corp

Inventors

• Kumar Sathish S B
• Sen Prantik

Classifications

• G06V30/422

  Technical drawings; Geographical maps · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N20/10

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title