Systems and methods for robotic process automation of mobile platforms

Patent metadata

Field	Value
Publication number	US-11656883-B2
Application number	US-202217658682-A
Country	US
Kind code	B2
Filing date	Apr 11, 2022
Priority date	May 28, 2020
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

In some embodiments, a robotic process automation (RPA) design application provides a user-friendly graphical user interface that unifies the design of automation activities performed on desktop computers with the design of automation activities performed on mobile computing devices such as smartphones and wearable computers. Some embodiments connect to a model device acting as a substitute for an actual automation target device (e.g. smartphone of specific make and model) and display a model GUI mirroring the output of the respective model device. Some embodiments further enable the user to design an automation workflow by directly interacting with the model GUI.

First claim

What is claimed is:

1. A method comprising employing at least one hardware processor of a computer system to execute a robotic process automation (RPA) design application, wherein executing the RPA design application comprises:
displaying a model graphical user interface (GUI) on a screen of the computer system, the model GUI mimicking a display of a mobile computing device targeted for automation;
receiving a user input indicating a user gesture for interacting with the mobile computing device;
receiving a user input selecting a target element of the model GUI; and
in response, outputting a computer-readable specification of an RPA robot configured to automatically carry out an operation that reproduces a result of applying the user gesture to another instance of the target element displayed by the mobile computing device.

2. The method of claim 1 , wherein executing the RPA application further comprises, in preparation for displaying the model GUI:
establishing a connection with an RPA model device sharing a set of hardware and software characteristics with the mobile computing device; and
retrieving a content of the model GUI from the RPA model device.

3. The method of claim 2 , wherein the RPA model device comprises a software emulation of the mobile computing device.

4. The method of claim 2 , wherein the content of the model GUI comprises a UI tree comprising a hierarchical representation of a plurality of UI elements of the model GUI including the target element.

5. The method of claim 2 , wherein executing the RPA design application further comprises, in preparation for establishing the connection with the RPA model device:
receiving a user input indicating a device type of the mobile computing device; and
in response, selecting the RPA model device from a plurality of available RPA model devices according to the device type.

6. The method of claim 5 , wherein:
executing the RPA design application further comprises displaying a plurality of device aliases, each device alias representing a distinct type of mobile computing device; and
the user input indicating the device type comprises the user selecting a device alias from the plurality of device aliases.

7. The method of claim 1 , wherein executing the RPA application further comprises displaying an RPA activity menu for selection by the user, the RPA activity menu comprising:
a first subset of menu items indicative of RPA activities for interacting with mobile computing devices, the first subset of menu items including an item indicative of the user gesture;
a second subset of menu items indicative of RPA activities for interacting with desktop computer systems; and
wherein the user input indicating the user gesture comprises the user selecting the item from the RPA activity menu.

8. The method of claim 1 , wherein executing the RPA application further comprises:
receiving a user input indicating an RPA activity for interacting with a desktop computer;
in response, displaying a desktop UI on the screen of the computer system, the desktop UI exposed by an RPA target application executing on the computer system;
in response to displaying the desktop UI, receiving a user input selecting a target desktop UI element from a plurality of UI elements of the desktop UI; and
formulating the computer-readable specification of the RPA robot further according to the RPA activity and target desktop UI element.

9. The method of claim 1 , wherein executing the RPA application further comprises displaying a graphical representation of an RPA workflow comprising a user-provided sequence of RPA activities including the user gesture, wherein:
the sequence of RPA activities is graphically divided into a plurality of activity containers, wherein one container exclusively comprises RPA activities configured to be performed on the mobile computing device, and another container exclusively comprises RPA activities configured to be performed on a desktop computer; and
the computer-readable specification of the RPA robot is formulated according to the sequence of RPA activities.

10. The method of claim 1 , wherein executing the RPA application further comprises:
recording a sequence of user actions performed by the user on the model GUI, the sequence of user actions including the user gesture;
in response to recording the sequence of user actions, displaying a graphical representation of an RPA workflow including a distinct graphical element for each user action of the sequence of user actions; and
formulating the computer-readable specification of an RPA robot according to the sequence of user actions.

11. The method of claim 1 , wherein the user gesture comprises an item selected from a group consisting of a touchscreen swipe gesture, a touchscreen pinch gesture, a touchscreen tap gesture, and a gesture that changes a spatial orientation of a mobile device.

12. A computer system comprising at least one hardware processor configured to execute a robotic process automation (RPA) design application, wherein executing the RPA design application comprises:
displaying a model graphical user interface (GUI) on a screen of the computer system, the model GUI mimicking a display of a mobile computing device targeted for automation;
receiving a user input indicating a user gesture for interacting with the mobile computing device;
receiving a user input selecting a target element of the model GUI; and
in response, outputting a computer-readable specification of an RPA robot configured to automatically carry out an operation that reproduces a result of applying the user gesture to another instance of the target element displayed by the mobile computing device.

13. The computer system of claim 12 , wherein executing the RPA application further comprises, in preparation for displaying the model GUI:
establishing a connection with an RPA model device sharing a set of hardware and software characteristics with the mobile computing device; and
retrieving a content of the model GUI from the RPA model device.

14. The computer system of claim 13 , wherein the RPA model device comprises a software emulation of the mobile computing device.

15. The computer system of claim 13 , wherein the content of the model GUI comprises a UI tree comprising a hierarchical representation of a plurality of UI elements of the model GUI including the target element.

16. The computer system of claim 13 , wherein executing the RPA design application further comprises, in preparation for establishing the connection with the RPA model device:
receiving a user input indicating a device type of the mobile computing device; and
in response, selecting the RPA model device from a plurality of available RPA model devices according to the device type.

17. The computer system of claim 16 , wherein:
executing the RPA design application further comprises displaying a plurality of device aliases, each device alias representing a distinct type of mobile computing device; and
the user input indicating the device type comprises the user selecting a device alias from the plurality of device aliases.

18. The computer system of claim 12 , wherein executing the RPA application further comprises displaying an RPA activity menu for selection by the user, the RPA activity menu comprising:
a first subset of menu items indicative of RPA activities for interacting with mobile computing devices, the first subset of menu items including an item indicative of the user gesture;
a second subset of menu items indicative of RPA activities for inte

Assignees

• Uipath Inc

Inventors

• Mayer Christian
• Weishaar Gerd
• Cucosel Bogdan N

Classifications

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G05B2219/40393

  Learn natural high level command, associate its template with a plan, sequence · CPC title

• G06N3/008

  based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour · CPC title

• G06F9/452 Primary

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title