Using artificial intelligence to iteratively design a user interface through progressive feedback

Patent metadata

Field	Value
Publication number	US-11650717-B2
Application number	US-201916507241-A
Country	US
Kind code	B2
Filing date	Jul 10, 2019
Priority date	Jul 10, 2019
Publication date	May 16, 2023
Grant date	May 16, 2023

Abstract

A computer-implemented method, system and computer program product for generating a user interface. A sketch (e.g., wireframe) of a portion of a user interface is received. The sketch is analyzed to predict a set of intended sketches using artificial intelligence based on historical data and/or the user's asset library. A set of intended final sketch renderings of the user interface is then generated and displayed using the set of predicted intended sketches based on historical data or a model trained to extract visual characteristics from existing user interface screens. If the user selects one of the intended final sketch renderings of the user interface as being directed to the intended design of the user interface and indicates that the selected intended final sketch rendering of the user interface corresponds to the final intended design, then code is generated to render the selected final sketch rendering of the user interface.

First claim

The invention claimed is:

1. A computer-implemented method for generating a user interface through progressive feedback, the method comprising:
receiving a sketch of a portion of a user interface, wherein said sketch is a wireframe;
analyzing said sketch to predict a first set of intended sketches of said user interface using artificial intelligence based on historical data consisting of renderings of user interface designs, wherein said historical data comprises previously rendered user interface screens of existing software applications;
generating and displaying a first set of intended final sketch renderings of said user interface using said first set of predicted intended sketches of said user interface based on a model trained to extract visual characteristics from existing user interface screens;
receiving a selection of a first final sketch rendering of said user interface from said first set of intended final sketch renderings of said user interface; and
generating code to render said selected first final sketch rendering of said user interface in response to a user indicating said selected first final sketch rendering of said user interface is a final intended design.

2. The method as recited in claim 1 further comprising:
generating and displaying a first set of keyframes to include design variations between said sketch and said selected first final sketch rendering of said user interface in response to said user not indicating said selected first final sketch rendering of said user interface is said final intended design, wherein said first set of keyframes are various options in designing said selected first final sketch rendering of said user interface based on said sketch, wherein each of said first set of keyframes has one or more unfinished portions.

3. The method as recited in claim 2 further comprising:
receiving a selection of one of said first set of keyframes comprising at least one unfinished portion;
receiving a subsequent sketch of said user interface that includes strokes within said at least one unfinished portion of said selected keyframe;
analyzing said subsequent sketch of said user interface to predict a second set of intended sketches of said user interface using artificial intelligence;
generating and displaying a second set of intended final sketch renderings of said user interface using said second set of predicted intended sketches of said user interface;
receiving a selection of a second final sketch rendering of said user interface from said second set of intended final sketch renderings of said user interface; and
generating code to render said selected second final sketch rendering of said user interface in response to said user indicating said selected second final sketch rendering of said user interface is said final intended design.

4. The method as recited in claim 1 , wherein said first set of intended final sketch renderings of said user interface is determined based on matching previously rendered user interfaces with elements closest in appearance to said first set of predicted intended sketches of said user interface.

5. The method as recited in claim 1 , wherein said model comprises an encoder/decoder model that is trained on information from user interface metadata and screenshots of user interfaces to translate said screenshots into a domain specific language and then into code.

6. The method as recited in claim 1 , wherein said model is trained on information from user interface component metadata and screenshots of user interfaces to translate said screenshots into a domain specific language and then into code, wherein said model is trained after classification of user interface components.

7. The method as recited in claim 1 , wherein said sketch is a continuation of a prior sketch of said user interface.

8. A computer program product for generating a user interface through progressive feedback, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code comprising the programming instructions for:
receiving a sketch of a portion of a user interface, wherein said sketch is a wireframe;
analyzing said sketch to predict a first set of intended sketches of said user interface using artificial intelligence based on historical data consisting of renderings of user interface designs, wherein said historical data comprises previously rendered user interface screens of existing software applications;
generating and displaying a first set of intended final sketch renderings of said user interface using said first set of predicted intended sketches of said user interface based on a model trained to extract visual characteristics from existing user interface screens;
receiving a selection of a first final sketch rendering of said user interface from said first set of intended final sketch renderings of said user interface; and
generating code to render said selected first final sketch rendering of said user interface in response to a user indicating said selected first final sketch rendering of said user interface is a final intended design.

9. The computer program product as recited in claim 8 , wherein the program code further comprises the programming instructions for:
generating and displaying a first set of keyframes to include design variations between said sketch and said selected first final sketch rendering of said user interface in response to said user not indicating said selected first final sketch rendering of said user interface is said final intended design, wherein said first set of keyframes are various options in designing said selected first final sketch rendering of said user interface based on said sketch, wherein each of said first set of keyframes has one or more unfinished portions.

10. The computer program product as recited in claim 9 , wherein the program code further comprises the programming instructions for:
receiving a selection of one of said first set of keyframes comprising at least one unfinished portion;
receiving a subsequent sketch of said user interface that includes strokes within said at least one unfinished portion of said selected keyframe;
analyzing said subsequent sketch of said user interface to predict a second set of intended sketches of said user interface using artificial intelligence;
generating and displaying a second set of intended final sketch renderings of said user interface using said second set of predicted intended sketches of said user interface;
receiving a selection of a second final sketch rendering of said user interface from said second set of intended final sketch renderings of said user interface; and
generating code to render said selected second final sketch rendering of said user interface in response to said user indicating said selected second final sketch rendering of said user interface is said final intended design.

11. The computer program product as recited in claim 8 , wherein said first set of intended final sketch renderings of said user interface is determined based on matching previously rendered user interfaces with elements closest in appearance to said first set of predicted intended sketches of said user interface.

12. The computer program product as recited in claim 8 , wherein said model comprises an encoder/decoder model that is trained on information from user interface metadata and screenshots of user interfaces to translate said screenshots into a domain specific language and then into code.

13. The computer program product as recited in claim 8 , wherein said model is trained on information from user interface co

Assignees

• IBM

Inventors

• Liu Eric
• Jiang Shun
• Megahed Aly
• Huang Lei
• Yin Peifeng
• Arar Raphael
• Ren Guangjie

Classifications

• G06F3/0484 Primary

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F8/38 Primary

  for implementing user interfaces · CPC title

• G06F8/35

  model driven · CPC title

• G06T11/20

  Drawing from basic elements · CPC title