Self-learning robotic process automation

Patent metadata

Field	Value
Publication number	US-10970090-B2
Application number	US-201916292602-A
Country	US
Kind code	B2
Filing date	Mar 5, 2019
Priority date	Jun 23, 2017
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a self-learning robotic process automation are disclosed. In one aspect, a method includes receiving an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface. The method further includes executing a command of the one or more commands of the automated script. The method further includes determining that an error occurred during execution of the command of the one or more commands of the automated script. The method further includes determining a modification for the command by applying a script repair model. The method further includes executing the modified command. The method further includes determining whether the error or another error occurred during execution of the modified command. The method further includes determining whether to update the automated script with the modified command.

First claim

What is claimed is:

1. A computer-implemented method comprising:
providing, for execution and by the computing device, a command of an automated script that is configured to (i) use computer vision to identify, in a graphical interface, a first group of pixels that match an image and (ii) interact with the first group of pixels;
determining, by the computing device, that an error occurred during execution of the command of the automated script;
in response to determining that the error occurred during execution of the command of the automated script, modifying, by the computing device, the command of the automated script by adjusting the image;
providing, for execution and by the computing device, the adjusted command of the automated script that is configured to (i) use computer vision to identify, in the graphical interface, a second group of pixels that match the adjusted image and (ii) interact with the second group of pixels;
determining, by the computing device, that the error or another error did not occur during execution of the automated script; and
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device, the command of the automated script using the adjusted image.

2. The method of claim 1 , wherein modifying the command of the automated script by adjusting the image comprises cropping the image.

3. The method of claim 1 , comprising:
determining that the graphical interface does not include the first group of pixels that match the image,
wherein modifying the command of the automated script is based on determining that the graphical interface does not include the first group of pixels that match the image.

4. The method of claim 1 , wherein determining, by the computing device, that an error occurred during execution of the command of the automated script comprises:
determining that the graphical interface does not include the first group of pixels.

5. The method of claim 1 , comprising:
in response to determining that the error occurred during execution of the command of the automated script, providing, by the computing device, the image and a representation of the graphical interface as an input to a script repair model that is configured to determine an adjustment to the image; and
receiving, from the script repair model, data indicating the adjustment to the image,
wherein adjusting the image is based on the data indicating the adjustment to the image.

6. The method of claim 5 , comprising:
receiving, by the computing device, one or more additional automated scripts that each include commands and results that correspond to each command; and
training, using machine learning, the script repair model using the one or more additional automated scripts that each include commands and the results that correspond to each command.

7. The method of claim 5 , comprising:
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device and using machine learning, the script repair model using the adjusted command and the adjusted image.

8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
providing, for execution and by the computing device, a command of an automated script that is configured to (i) use computer vision to identify, in a graphical interface, a first group of pixels that match an image and (ii) interact with the first group of pixels;
determining, by the computing device, that an error occurred during execution of the command of the automated script;
in response to determining that the error occurred during execution of the command of the automated script, modifying, by the computing device, the command of the automated script by adjusting the image;
providing, for execution and by the computing device, the adjusted command of the automated script that is configured to (i) use computer vision to identify, in the graphical interface, a second group of pixels that match the adjusted image and (ii) interact with the second group of pixels;
determining, by the computing device, that the error or another error did not occur during execution of the automated script; and
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device, the command of the automated script using the adjusted image.

9. The system of claim 8 , wherein modifying the command of the automated script by adjusting the image comprises cropping the image.

10. The system of claim 8 , wherein the operations comprise:
determining that the graphical interface does not include the first group of pixels that match the image,
wherein modifying the command of the automated script is based on determining that the graphical interface does not include the first group of pixels that match the image.

11. The system of claim 8 , wherein determining, by the computing device, that an error occurred during execution of the command of the automated script comprises:
determining that the graphical interface does not include the first group of pixels.

12. The system of claim 8 , wherein the operations comprise:
in response to determining that the error occurred during execution of the command of the automated script, providing, by the computing device, the image and a representation of the graphical interface as an input to a script repair model that is configured to determine an adjustment to the image; and
receiving, from the script repair model, data indicating the adjustment to the image,
wherein adjusting the image is based on the data indicating the adjustment to the image.

13. The system of claim 12 , wherein the operations comprise:
receiving, by the computing device, one or more additional automated scripts that each include commands and results that correspond to each command; and
training, using machine learning, the script repair model using the one or more additional automated scripts that each include commands and the results that correspond to each command.

14. The system of claim 12 , wherein the operations comprise:
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device and using machine learning, the script repair model using the adjusted command and the adjusted image.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
providing, for execution and by the computing device, a command of an automated script that is configured to (i) use computer vision to identify, in a graphical interface, a first group of pixels that match an image and (ii) interact with the first group of pixels;
determining, by the computing device, that an error occurred during execution of the command of the automated script;
in response to determining that the error occurred during execution of the command of the automated script, modifying, by the computing device, the command of the automated script by adjusting the image;
providing, for execution and by the computing device, the adjusted command of the automated script that is configured to (i) use computer vision to identify, in the graphical interface, a second group of pixels that match th

Assignees

• Accenture Global Solutions Ltd

Inventors

• Hall Simon
• Schiopu Vitalie
• Van Den Heuvel Jeroen
• Jacquot Adrien

Classifications

• G06F11/3672

  Test management · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F9/451 Primary

  Execution arrangements for user interfaces · CPC title

• G06F11/0715

  in a system implementing multitasking (multitasking per se G06F9/46) · CPC title

• G06F11/0793

  Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40) · CPC title