Robotic process automation with resilient playback capabilities

Patent metadata

Field	Value
Publication number	US-11775321-B2
Application number	US-202017139838-A
Country	US
Kind code	B2
Filing date	Dec 31, 2020
Priority date	Aug 3, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

Robotic process automation (RPA) systems with improved playback capabilities. Certain embodiments can provide resilient playback of software automation processes by providing enhanced capability to locate user interface controls within a screen image of a user interface. In some embodiments, one or more of the user interface controls located within the screen image of the user interface can be used to manipulate the user interface so that other user interface controls become visible within the screen image. Other embodiments can, additionally or alternatively, provide resilient playback of software automation processes by providing the capability to record, compute and store parameters for user interface controls detected from a screen image of a user interface. These parameters can be used to assist in locating correct corresponding user interface controls within a screen image presented at playback of a software automation process. Advantageously, embodiments disclosed herein allow software automation processes to operate with greater reliability and flexibility.

First claim

What is claimed is:

1. A computer-implemented method for facilitating resilient execution of a robotic process automation, the method comprising:
capturing a first image of a playback user interface that is presented on a display device associated with a playback computing device;
detecting at least one user interface control within the captured first image of the playback user interface, the detected user interface control being able to alter visible content of the playback user interface that is presented on the display device associated with the playback computing device;
effecting manipulation of the playback user interface using the detected user interface control to alter the playback user interface, whereby visible content of the altered playback user interface being presented is at least partially different than visible content the playback user interface presented before the effecting manipulation;
capturing a second image of the altered playback user interface that is presented on the display device associated with the playback computing device; and
detecting one or more user interface controls within the captured second image of the altered playback user interface.

2. A computer-implemented method as recited in claim 1 , wherein the effecting manipulation effects visual appearance and displayed content of the playback user interface.

3. A computer-implemented method as recited in claim 1 , wherein the method comprises:
selecting at least one of the detected user interface controls within the captured second image of the altered playback user interface; and
effecting a programmatic interaction with the selected at least one of the detected user interface controls within the captured second image of the altered playback user interface.

4. A computer-implemented method as recited in claim 3 , wherein the programmatic interaction carries out a portion of the robotic automation process.

5. A computer-implemented method as recited in claim 3 , wherein the robotic automation process performs a software automation process derived from a plurality of previously recorded user interactions with one or more application programs, and wherein the programmatic interaction interacts with the one or more application programs to carry out a portion of the software automation process.

6. A computer-implemented method as recited in claim 1 , wherein the detected at least one user interface control within the captured first image of the playback user interface pertains to a slider bar control.

7. A computer-implemented method as recited in claim 1 , wherein the method comprises:
determining movement capability of the detected at least one user interface control within the captured first image of the playback user interface; and
determining a movement amount for the effecting the manipulation of the playback user interface using the detected user interface control.

8. A computer-implemented method as recited in claim 7 , wherein the effecting manipulation of the playback user interface using the detected user interface control is done in accordance with the determined movement amount.

9. A computer-implemented method as recited in claim 8 , wherein the determined movement amount is a scroll amount.

10. A computer-implemented method as recited in claim 8 , wherein the detected at least one user interface control within the captured first image of the playback user interface pertains to a slider bar control.

11. A computer-implemented method as recited in claim 1 , wherein the detecting of the at least one user interface control within the captured first image of the playback user interface comprises determining parameters of the detected at least one user interface control within the captured first image of the playback user interface.

12. A computer-implemented method as recited in claim 1 , wherein the detecting of the at least one user interface control within the captured first image of the playback user interface comprises identifying at least one child control of the detected at least one user interface control within the captured first image of the playback user interface.

13. A computer-implemented method as recited in claim 1 , wherein the method comprises:
determining whether a particular user interface control requested by the robotic automation process is one of the one or more user interface controls within the captured second image of the altered playback user interface.

14. A computer-implemented method as recited in claim 13 , wherein the method comprises:
effecting a programmatic interaction with the particular user interface control in accordance with the robotic automation process if it is determined that the particular user interface control requested by the robotic automation process is one of the one or more user interface controls within the captured second image of the altered playback user interface.

15. A computer-implemented method for facilitating robotic process automation, the method comprising:
identifying a software automation process that facilitates automation of a previously captured series of user interactions with at least one user interface presented to a user on a display device associated with a capture computing device;
initiating execution of the software automation process;
capturing an image of a playback user interface that is presented on a display device associated with a playback computing device;
examining the captured image of the playback user interface to determine whether a corresponding user interface control is found within the captured image of the playback user interface;
effecting programmatic user interaction with the corresponding user interface control of the playback user interface when the examining determines that the captured image of the playback user interface includes the corresponding user interface control; and
manipulating the playback user interface when the examining determines that the captured image of the playback user interface does not include the corresponding user interface control, the manipulating of the playback user interface at least alters the playback user interface such that visible content of the altered playback user interface when presented is at least partially different than visible content the playback user interface presented before the manipulating.

16. A computer-implemented method as recited in claim 15 , wherein the manipulating comprises:
examining the captured image of the playback user interface to determine whether a movement user interface control is found within the captured image of the playback user interface; and
effecting manipulation of the playback user interface using the movement user interface control to alter the playback user interface if the examining of the captured image of the playback user interface determines that the movement user interface control is found within the captured image of the playback user interface.

17. A computer-implemented method as recited in claim 16 , wherein the method comprises:
capturing a subsequent image of the altered playback user interface that is presented on the display device associated with the playback computing device;
examining the captured subsequent image of the altered playback user interface to determine whether the corresponding user interface control is found within the captured subsequent image of the altered playback user interface; and
effecting programmatic user interaction with the corresponding user interface control of the altered playback user interface if the examining determines that the captured

Assignees

• Automation Anywhere Inc

Inventors

• Singh Sudhir Kumar
• Truscott Jesse
• Anand Virinchipuram
• Lodhiya Harshil

Classifications

• G06F9/451 Primary

  Execution arrangements for user interfaces · CPC title

• B25J13/06

  Control stands, e.g. consoles, switchboards · CPC title

• G06F3/0485

  Scrolling or panning · CPC title

• G05B2219/32128

  Gui graphical user interface · CPC title

• G05B2219/50391

  Robot · CPC title