Artificial intelligence / machine learning model drift detection and correction for robotic process automation

Patent metadata

Field	Value
Publication number	US-12321876-B2
Application number	US-202016934369-A
Country	US
Kind code	B2
Filing date	Jul 21, 2020
Priority date	Jul 21, 2020
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

Artificial intelligence (AI)/machine learning (ML) model drift detection and correction for robotic process automation (RPA) is disclosed. Information is analyzed pertaining to input data for an AI/ML model to determine whether data drift has occurred, analyze information pertaining to results from execution of the AI/ML model to determine whether model drift has occurred, or both. When, based on the analysis of the information, a change condition is found, a change threshold is met or exceeded, or both, the AI/ML model is retrained. The retrained AI/ML model may then be deployed to provide better predictions on real world data.

First claim

The invention claimed is:

1. A non-transitory computer-readable medium storing a computer program for performing artificial intelligence (AI)/machine learning (ML) model drift detection and correction for robotic process automation (RPA), the computer program configured to cause at least one processor to:
analyze information pertaining to input data for an AI/ML model executed by one or more of RPA robots to determine whether data drift has occurred, analyze information pertaining to results from execution of the AI/ML model to determine whether model drift has occurred, or both, wherein the information comprises one or more statistical distributions of predictions made by the AI/ML model;
map the one or more statistical distributions of the predictions made by the AI/ML model to respective actions taken by the one or more RPA robots in respective activities of RPA workflows; and
responsive to, based on the analysis of the information, a change condition being found, a change threshold is met or exceeded, or both, retrain the AI/ML model.

2. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
verify that the change condition, the change threshold, or both, have been met or exceeded for at least a predetermined period of time.

3. The non-transitory computer-readable medium of claim 1 , wherein the analysis comprises measuring one or more distances between unidimensional distributions of an original dataset and a new dataset for covariate drift.

4. The non-transitory computer-readable medium of claim 1 , wherein the analysis comprises determining whether a statistical moment changes.

5. The non-transitory computer-readable medium of claim 4 , wherein the statistical moment comprises a mean, a variance, a skewness, a kurtosis, a covariance, or a combination thereof.

6. The non-transitory computer-readable medium of claim 1 , wherein the information comprises what predictions the AI/ML model made, how many predictions the AI/ML model made and how many predictions were used by RPA robots, one or more statistical distributions of the predictions made by the AI/ML model, input data that was provided to the AI/ML model, or a combination thereof.

7. The non-transitory computer-readable medium of claim 1 , the computer program further configured to cause the at least one processor to:
verify that the retrained AI/ML model meets one or more performance thresholds; and
responsive to the retrained AI/ML model meeting the one or more performance thresholds, deploy the retrained AI/ML model.

8. The non-transitory computer-readable medium of claim 7 , wherein the retrained AI/ML model is deployed with a previous version of the AI/ML model and the computer program is further configured to cause the at least one processor to:
measure model drift of the retrained AI/ML model and the previous version of the AI/ML model using real data; and
responsive to the retrained AI/ML model performing better on the real data than the previous version of the AI/ML model based on the measured model drift, use the retrained AI/ML model in place of the previous version of the AI/ML model.

9. The non-transitory computer-readable medium of claim 8 , wherein responsive to the retrained AI/ML model not performing better on the real data than the previous version of the AI/ML model based on the measured model drift, the computer program is further configured to cause the at least one processor to:
discard the retrained AI/ML model or perform further retraining.

10. The non-transitory computer-readable medium of claim 7 , wherein the retrained AI/ML model is deployed using a canary testing technique or a champion/challenger testing technique.

11. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
receive a request to execute the AI/ML model and input information for the AI/ML model from an RPA robot;
verify that the input information conforms with a framework for the AI/ML model; and
responsive to the input information conforming with the framework:
execute the AI/ML model using the input information, and
return a result of the execution of the AI/ML model to the RPA robot.

12. The non-transitory computer-readable medium of claim 11 , wherein responsive to the input information not conforming with the framework, the computer program is further configured to cause the at least one processor to:
provide an error to the RPA robot indicating that the input information does not conform with the framework.

13. The non-transitory computer-readable medium of claim 11 , wherein the framework is enforced for a plurality of AI/ML models.

14. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
perform a dimensionality reduction technique on the information to generate a two-dimensional representation of the information; and
display the two-dimensional representation of the information.

15. The non-transitory computer-readable medium of claim 1 , wherein the analysis of the information is performed across multiple RPA processes executed by multiple respective RPA robots.

16. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
select the retrained AI/ML model or a previous version of the AI/ML model based on the input data; and
call the selected version of the AI/ML model.

17. The non-transitory computer-readable medium of claim 16 , wherein the selection is made in a hierarchical manner.

18. A computer-implemented method for performing artificial intelligence (AI)/machine learning (ML) model drift detection and correction for robotic process automation (RPA), comprising:
analyzing information pertaining to input data for an AI/ML model executed by the one or more RPA robots to determine whether data drift has occurred, analyzing information pertaining to results from execution of the AI/ML model to determine whether model drift has occurred, or both, wherein the information comprises one or more statistical distributions of predictions made by the AI/ML model;
mapping the one or more statistical distributions of the predictions made by the AI/ML model to respective actions taken by the one or more RPA robots in respective activities of RPA workflows; and
responsive to, based on the analysis of the information, a change condition being found, a change threshold is met or exceeded, or both:
retraining the AI/ML model,
verifying that the retrained AI/ML model meets one or more performance thresholds, and
responsive to the retrained AI/ML model meeting the one or more performance thresholds, deploying the retrained AI/ML model.

19. The computer-implemented method of claim 18 , further comprising:
verifying that the change condition, the change threshold, or both, have been met or exceeded for at least a predetermined period of time.

20. The computer-implemented method of claim 18 , wherein the retrained AI/ML model is deployed with a previous version of the AI/ML model and the method further comprises:
measuring model drift of the retrained AI/ML model and the previous version of the AI/ML model using real data; and
responsive to the retrained AI/ML model performing better on the real data than the previous version of the AI/ML model based on the measured mo

Assignees

• Uipath Inc

Inventors

• Singh Prabhdeep
• Mcgonnell Anton
• Hidalgo Marco Alban

Classifications

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G05B13/0265

  the criterion being a learning criterion · CPC title

• G06N20/00

  Machine learning · CPC title

• G05B13/026

  using a predictor · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title