Systems and methods for robotic process automation of mobile platforms

Patent metadata

Field	Value
Publication number	US-11941419-B2
Application number	US-202318312041-A
Country	US
Kind code	B2
Filing date	May 4, 2023
Priority date	May 28, 2020
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

In some embodiments, a robotic process automation (RPA) design application provides a user-friendly graphical user interface that unifies the design of automation activities performed on desktop computers with the design of automation activities performed on mobile computing devices such as smartphones and wearable computers. Some embodiments connect to a model device acting as a substitute for an actual automation target device (e.g., smartphone of specific make and model) and display a model GUI mirroring the output of the respective model device. Some embodiments further enable the user to design an automation workflow by directly interacting with the model GUI.

First claim

The invention claimed is:

1. A method comprising employing at least one hardware processor of a computer system to:
display a device management interface configured to receive a user input specifying a configuration of a mobile computing device targeted for automation;
in response to the user input, display a model graphical user interface (GUI) on a screen of the computer system, the model GUI mimicking a display of the mobile computing device; and
in response to another user input indicating a robotic process automation (RPA) activity for interacting with the mobile computing device:
update the model GUI to show a result of executing the RPA activity, and
output a computer-readable specification of an RPA robot configured to automatically execute the RPA activity on the mobile computing device.

2. The method of claim 1 , further comprising, in preparation for displaying the model GUI, employing the at least one hardware processor to:
establish a connection with an RPA model device sharing a set of hardware and software characteristics with the mobile computing device; and
retrieve a content of the model GUI from the RPA model device.

3. The method of claim 2 , wherein the RPA model device comprises a software emulation of the mobile computing device.

4. The method of claim 1 , wherein specifying the configuration of the mobile computing device comprises identifying a type of operating system of the mobile computing device.

5. The method of claim 1 , wherein the user input comprises selecting the mobile computing device from a plurality of candidate mobile computing devices, the plurality of candidate mobile computing devices comprising a first device executing a first type of operating system and a second device executing a second type of operating system.

6. The method of claim 1 , wherein the user input comprises selecting the mobile computing device from a plurality of candidate mobile computing devices, the plurality of candidate mobile computing devices comprising a first device from a first manufacturer and a second device from a second manufacturer.

7. The method of claim 1 , wherein:
specifying the configuration of the mobile computing device comprises identifying a software application executing on the mobile computing device, the software application targeted for automation; and
wherein the model GUI mimics a GUI exposed by the software application on the mobile computing device.

8. The method of claim 1 , wherein:
the at least one hardware processor is configured to display an activity menu listing a set of RPA activities for interacting with the mobile computing device; and
wherein the other user input comprises a user's selecting the RPA activity from the activity menu.

9. The method of claim 1 , wherein the RPA activity comprises a user gesture for interacting with the mobile computing device.

10. The method of claim 1 , wherein the at least one hardware processor is further configured to:
in response to displaying the model GUI, detect a user's interaction with the model GUI; and
automatically identify the RPA activity according to the user's interaction.

11. The method of claim 1 , wherein the at least one hardware processor is further configured to display a graphical representation of an RPA workflow comprising a user-provided sequence of RPA activities including the RPA activity, and wherein the RPA robot is configured to execute the sequence of RPA activities.

12. A computer system comprising at least one hardware processor configured to:
display a device management interface configured to receive a user input specifying a configuration of a mobile computing device targeted for automation;
in response to the user input, display a model graphical user interface (GUI) on a screen of the computer system, the model GUI mimicking a display of the mobile computing device; and
in response to another user input indicating a robotic process automation (RPA) activity for interacting with the mobile computing device:
update the model GUI to show a result of executing the RPA activity, and
output a computer-readable specification of an RPA robot configured to automatically execute the RPA activity on the mobile computing device.

13. The computer system of claim 12 , further comprising, in preparation for displaying the model GUI, employing the at least one hardware processor to:
establish a connection with an RPA model device sharing a set of hardware and software characteristics with the mobile computing device; and
retrieve a content of the model GUI from the RPA model device.

14. The computer system of claim 13 , wherein the RPA model device comprises a software emulation of the mobile computing device.

15. The computer system of claim 12 , wherein specifying the configuration of the mobile computing device comprises identifying a type of operating system of the mobile computing device.

16. The computer system of claim 12 , wherein the user input comprises selecting the mobile computing device from a plurality of candidate mobile computing devices, the plurality of candidate mobile computing devices comprising a first device executing a first type of operating system and a second device executing a second type of operating system.

17. The computer system of claim 12 , wherein the user input comprises selecting the mobile computing device from a plurality of candidate mobile computing devices, the plurality of candidate mobile computing devices comprising a first device from a first manufacturer and a second device from a second manufacturer.

18. The computer system of claim 12 , wherein:
specifying the configuration of the mobile computing device comprises identifying a software application executing on the mobile computing device, the software application targeted for automation; and
wherein the model GUI mimics a GUI exposed by the software application on the mobile computing device.

19. The computer system of claim 12 , wherein:
the at least one hardware processor is configured to display an activity menu listing a set of RPA activities for interacting with the mobile computing device; and
wherein the other user input comprises a user's selecting the RPA activity from the activity menu.

20. The computer system of claim 12 , wherein the RPA activity comprises a user gesture for interacting with the mobile computing device.

21. The computer system of claim 12 , wherein the at least one hardware processor is further configured to:
in response to displaying the model GUI, detect a user's interaction with the model GUI; and
automatically identify the RPA activity according to the user's interaction.

22. The computer system of claim 12 , wherein the at least one hardware processor is further configured to display a graphical representation of an RPA workflow comprising a user-provided sequence of RPA activities including the RPA activity, and wherein the RPA robot is configured to execute the sequence of RPA activities.

23. A non-transitory computer-readable medium storing instructions which, when executed by at least one hardware processor of a computer system, cause the computer system to:
display a device management interface configured to receive a user input specifying a configuration of a mobile computing device targeted for automation;
in response to the user input, display a model graphical user interface (GUI) on a screen of the computer system, the model GUI mimicking a display of the mobile comput

Assignees

• Uipath Inc

Inventors

• Mayer Christian
• Weishaar Gerd
• Cucosel Bogdan

Classifications

• G06F9/452 Primary

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F9/547

  Remote procedure calls [RPC]; Web services · CPC title

• G05B2219/40391

  Human to robot skill transfer · CPC title

• G05B2219/40392

  Programming, visual robot programming language · CPC title