Self-learning robotic process automation

Patent metadata

Field	Value
Publication number	US-10235192-B2
Application number	US-201715632099-A
Country	US
Kind code	B2
Filing date	Jun 23, 2017
Priority date	Jun 23, 2017
Publication date	Mar 19, 2019
Grant date	Mar 19, 2019

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a self-learning robotic process automation are disclosed. In one aspect, a method includes receiving an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface. The method further includes executing a command of the one or more commands of the automated script. The method further includes determining that an error occurred during execution of the command of the one or more commands of the automated script. The method further includes determining a modification for the command by applying a script repair model. The method further includes executing the modified command. The method further includes determining whether the error or another error occurred during execution of the modified command. The method further includes determining whether to update the automated script with the modified command.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a script executor of a script repair system that includes (i) the script executor, (ii) an error detector, (iii) a command modifier, and (iv) a modified command simulator, an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface on a display of a computer;
executing, by the script executor of the script repair system, a command of the one or more commands of the automated script;
during execution of the command of the one or more commands of the automated script, analyzing, using computer vision and by the script executor of the script repair system, pixels of a screen capture of the user interface on the display of the computer for pixels that match pixels of an image that corresponds to the command;
determining, by the error detector of the script repair system, that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command;
based on determining that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command, determining, by the error detector of the script repair system, that an error occurred during execution of the command of the one or more commands of the automated script;
in response to determining that an error occurred during execution of the command of the one or more commands of the automated script, cropping, by the command modifier of the script repair system, the image that corresponds to the command by removing a portion of the pixels of the image that corresponds to the command to match a group of the pixels of the screen capture of the user interface on the display of the computer;
executing, by the modified command simulator of the script repair system, the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
determining, by the error detector of the script repair system, that the error or another error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
based on determining that the error or the other error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command, updating, by the command modifier of the script repair system, the automated script with the cropped image; and
executing, by the script executor of the script repair system, the automated script with the cropped image.

2. The method of claim 1 , comprising:
based on determining that the error or the other error did not occur during execution of the modified command, updating a script repair model using data indicating that the error or the other error did not occur during execution of the modified command.

3. The method of claim 1 , comprising:
identifying another automated script that includes another image that is similar to the command of the automated script; and
updating the other image of the other automated script with the cropped image.

4. The method of claim 1 , comprising:
classifying, by a script repair model, the error that occurred during execution of the command of the one or more commands of the automated script; and
based on the classification of the error, determining a modification for the command by applying, to the command, the script repair model.

5. The method of claim 4 , comprising:
based on applying the script repair model to the command, determining for each of one or more candidate modifications, a repair confidence score that reflects a likelihood that applying a respective candidate modification to the command will correct the error; and
based on the repair confidence scores, selecting, from among the one or more candidate modifications, the modification for the command.

6. The method of claim 4 , comprising:
receiving one or more automated scripts that each include commands and results that correspond to each command; and
training, using machine learning and the one or more automated scripts that each include commands and results that correspond to each command, the script repair model.

7. The method of claim 1 , wherein:
executing the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command comprises:
simulating selection of the group of pixels of the screen capture, and

determining whether the error or another error occurred during execution of the modified command comprises:
determining whether the error or the other error occurred during simulation of the selection of the group of pixels of the screen capture.

8. A system comprising:
one or more computers; and
one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, by a script executor of a script repair system that includes (i) the script executor, (ii) an error detector, (iii) a command modifier, and (iv) a modified command simulator, an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface on a display of a computer;
executing, by the script executor of the script repair system, a command of the one or more commands of the automated script;
during execution of the command of the one or more commands of the automated script, analyzing, using computer vision and by the script executor of the script repair system, pixels of a screen capture of the user interface on the display of the computer for pixels that match pixels of an image that corresponds to the command;
determining, by the error detector of the script repair system, that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command;
based on determining that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command, determining, by the error detector of the script repair system, that an error occurred during execution of the command of the one or more commands of the automated script;
in response to determining that an error occurred during execution of the command of the one or more commands of the automated script, cropping, by the command modifier of the script repair system, the image that corresponds to the command by removing a portion of the pixels of the image that corresponds to the command to match a group of the pixels of the screen capture of the user interface on the display of the computer;
executing, by the modified command simulator of the script repair system, the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
determining, by the error detector of the script repair system, that the error or another error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
based on determining that the error or the other error did not occur during selec

Assignees

• Accenture Global Solutions Ltd

Inventors

• Hall Simon
• Schiopu Vitalie
• Van Den Heuvel Jeroen
• Jacquot Adrien

Classifications

• G06F9/451 Primary

  Execution arrangements for user interfaces · CPC title

• G06F11/3672

  Test management · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06N99/005

  Physics · mapped topic

• G06F11/3696

  Methods or tools to render software testable · CPC title