Automatically locating elements in user interfaces

Patent metadata

Field	Value
Publication number	US-11475205-B2
Application number	US-202016779390-A
Country	US
Kind code	B2
Filing date	Jan 31, 2020
Priority date	Jan 31, 2020
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Techniques are disclosed relating to determining locations of child elements in a user interface (UI), automatically updating the locations of the child elements in the UI, and generating an updated UI with the updated locations. A computer system may receive information specifying content of the UI and render an image of the UI based on the received information. Computer vision algorithms may be applied to the rendered of the UI to determine locations of elements in the UI. The locations of the elements may be updated to position the elements in selected locations relative to each other. The updated locations of the elements as determined by the computer vision algorithms may be implemented to generate updating information specifying content for the updated UI.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a computer system, information specifying content of a user interface (UI);
generating, by the computer system based on the received information, a rendered image of the UI, wherein the rendered image includes a first rendered parent element in the UI, and wherein the rendered parent element includes a rendering of at least two child elements having content;
identifying, by applying a machine learning algorithm to the rendered image, the first rendered parent element;
determining, by applying the machine learning algorithm to the rendered parent element in the rendered image, location information for the content included within the at least two child elements, wherein the location information includes pixel-based determinations of locations of the content inside the rendered parent element;
accessing, by the computer system, location rules for the UI, wherein the location rules include at least one alignment rule for content in the UI;
updating, by the computer system, for the locations of the content inside the rendered parent element until the content inside the rendered parent element is positioned in alignment according to the at least one alignment rule; and
generating, by the computer system, updated information specifying the content of the UI based on the locations of the content inside the rendered parent element when the content is positioned in alignment according to the at least one alignment rule.

2. The method of claim 1 , wherein determining the location information for the content included within the at least two child elements includes determining bounding regions for the content inside the rendered parent element that is visible in the rendered image and determining location information for the bounding regions.

3. The method of claim 2 , wherein the determined location information includes spacing information for the bounding regions inside the rendered parent element.

4. The method of claim 2 , wherein the bounding regions for the content inside the rendered parent element that is visible in the rendered image are determined independently of bounding regions for the at least two child elements defined by the information specifying content of the UI.

5. The method of claim 1 , further comprising determining, by the application of the machine learning algorithm on the rendered image, identifying information of the content inside the rendered parent element, wherein determining the identification information includes determining a type of content for the content inside the rendered parent element.

6. The method of claim 5 , wherein determining the type of content includes determining whether the content is text content, image content, or shape content.

7. The method of claim 5 , wherein at least one child element is an image element, and wherein determining the type of content included within the image element includes identifying image content and determining a confidence level for the identification of the image content.

8. The method of claim 1 , wherein the machine learning algorithm is a computer vision algorithm.

9. The method of claim 1 , further comprising generating, by the computer system, a report of the updated locations of the content inside the rendered parent element when the content is positioned in alignment according to the at least one alignment rule.

10. The method of claim 1 , wherein updating the locations of the content inside the rendered parent element until the content inside the rendered parent element are positioned in alignment according to the at least one alignment rule includes:
(a) updating the locations of the content inside the rendered parent element;
(b) determining, by applying the machine learning algorithm to the rendered parent element, the updated location information for the content inside the rendered parent element;
determining, by the computer system, whether the content inside the rendered parent element is positioned in alignment according to the at least one alignment rule based on the updated location information;
in response to the content included inside the rendered parent element being determined to be positioned in alignment according to the at least one alignment rule, generating the updated information specifying the content of the UI based on the updated location information for the content inside the rendered parent element; and
in response to the content inside the rendered parent element being determined to not be positioned in the alignment according to the at least one alignment rule, repeating (a) and (b) until the content inside the rendered parent element is determined to be positioned in alignment according to the at least one alignment rule.

11. The method of claim 1 :
wherein determining the location information for the content included within the at least two child elements includes determining first location information for first content included within a first child element of the at least two child elements and second location information for second content included within a second child element of the at least two child elements, wherein the first location information and the second location information include pixel-based determinations of locations of the first content and the second content inside the at least two child elements; and
wherein updating locations of the content inside the rendered parent element includes updating a location for the first content inside the rendered parent element with respect to a location for the second content inside the rendered parent element.

12. A non-transitory computer-readable medium having program instructions stored thereon that are executable by a user interface (UI) system to perform operations comprising:
receiving, by a computer system, information specifying content of a user interface (UI);
generating, by the computer system based on the received information, a rendered image of the UI, wherein the rendered image includes a first rendered parent element in the UI, and wherein the rendered parent element includes a rendering of at least two child elements having content;
identifying, by applying a machine learning algorithm to the rendered image, the first rendered parent element;
determining, by applying the machine learning algorithm to the rendered parent element in the rendered image, location information for the content included within the at least two child elements, wherein the location information includes pixel-based determinations of locations of the content inside the rendered parent element;
accessing, by the computer system, location rules for the UI, wherein the location rules include at least one alignment rule for content in the UI;
updating, by the computer system, for the locations of the content inside the rendered parent element until the content inside the rendered parent element is positioned in alignment according to the at least one alignment rule; and
generating, by the computer system, updated information specifying the content of the UI based on the locations of the content inside the rendered parent element when the content is positioned in alignment according to the at least one alignment rule; and
generating, by the computer system, a report of the updated locations of the content inside the rendered parent element.

13. The non-transitory computer-readable medium of claim 12 , further comprising transmitting, by the computer system, the report to a computer system operating the UI.

14. The non-transitory computer-readable medium of claim 12 , further comprising transmitting, by the computer syst

Assignees

• Salesforce Com Inc

Inventors

• Lundin Jessica
• Woodward David James
• Ross Alan Martin
• Lonsdorf Brian J
• Schoppe Owen Winne
• Rohde Sönke

Classifications

• G06F40/151

  Transformation · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F40/106 Primary

  Display of layout of documents; Previewing · CPC title

• G06F8/34

  Graphical or visual programming · CPC title