Computer-automated processing with rule-supplemented machine learning

Patent metadata

Field	Value
Publication number	US-12367403-B2
Application number	US-202117332223-A
Country	US
Kind code	B2
Filing date	May 27, 2021
Priority date	May 27, 2021
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

Various examples are directed to systems and methods for executing a computer-automated process using trained machine learning (ML) models. A computing system may access first event data describing a first event. The computing system may execute a first ML model to determine an ML characterization of the first event using the first event data. The computing system may also apply a first rule set to the first event data to generate a rule characterization of the first event. The computing system may determine an output characterization of the first event based at least in part on the rule characterization of the first event and determine to deactivate the first rule set based at least in part on the ML characterization of the first event.

First claim

What is claimed is:

1. A method of executing a computer-automated process using trained machine learning (ML) models, the method comprising:
using at least one rule set to determine that a first ML model is untrained to characterize a first event type;
modifying the first ML model;
accessing first event data describing a first event of the first event type;
after modifying the first ML model, executing, by one or more hardware processors, the first ML model to determine an ML characterization of the first event using the first event data;
applying, by the one or more hardware processors, a first rule set to the first event data to generate a rule characterization of the first event;
determining, by the one or more hardware processors, an output characterization of the first event based at least in part on the rule characterization of the first event; and
determining, by the one or more hardware processors, to deactivate the first rule set based at least in part on the ML characterization of the first event.

2. The method of claim 1 , further comprising, before applying the first rule set to the first event data, applying a proactive detection rule set to the first event data to determine that the first event is an untrained event.

3. The method of claim 1 , further comprising:
accessing second event data describing a second event of a second event type different than the first event type;
executing the first ML model to determine an ML characterization of the second event using the second event data;
determining that a confidence of the ML characterization of the second event is less than a threshold level;
applying a reactive detection rule set to the first event data to determine that the second event is not an untrained event; and
responsive to determining that the second event is not an untrained event, sending an indication of the second event to a manual processing queue.

4. The method of claim 1 , further comprising:
accessing second event data describing a second event of a second event type different than the first event type;
executing the first ML model to determine an ML characterization of the second event using the second event data;
determining that a confidence of the ML characterization of the second event is less than a confidence threshold; and
applying a reactive detection rule set to the second event data to determine that the second event is an untrained event, the applying of the first rule set being responsive to the determining that the first event is an untrained event.

5. The method of claim 4 , further comprising:
receiving third event data describing a third event for characterization;
executing the first ML model to determine an ML characterization of the third event using the third event data;
determining that a confidence of the ML characterization of the third event is less than the confidence threshold;
applying the reactive detection rule set to the third event data to determine that the third event is not an untrained event; and
directing the third event to a manual exception process.

6. The method of claim 5 , the determining to deactivate the first rule set being based at least in part on a ratio of events determined to be untrained events using the reactive detection rule set to events determined not to be untrained events using the reactive detection rule set.

7. The method of claim 1 , the ML characterization comprising a first score for the first event, the rule characterization comprising a supplementary value, the method further comprising combining the supplementary value and the ML characterization to determine the output characterization of the first event.

8. The method of claim 1 , the first ML model being a training ML model retrained using training data derived from a plurality of untrained events, the method further comprising:
executing a deployed ML model to generate a deployed ML model characterization of the first event, the deployed ML model characterization comprising a score for the first event; and
determining the output characterization of the first event at least in part by combining the rule characterization of the first event and the deployed ML model characterization of the first event.

9. The method of claim 8 , the determining to deactivate the first rule set comprising determining that a difference between the ML characterization of the first event and a desired characterization of the first event is less than a first threshold.

10. The method of claim 8 , further comprising:
determining that a difference between the ML characterization of the first event and a desired characterization of the first event is less than a first threshold;
responsive to determining that the difference between the ML characterization of the first event and the desired characterization of the first event is less than a first threshold, deploying the first ML model;
training a second version of the first ML model; and
determining that a difference between at least one ML characterization generated by the second version of the first ML model and the desired characterization is less than a second threshold, the second threshold being smaller than the first threshold.

11. A system for executing a computer-automated process using trained machine learning (ML) models, the system comprising:
at least one computing device comprising a processor and a memory, the at least one computing device being programmed to perform operations comprising:
using at least one rule set to determine that a first ML model is untrained to characterize a first event type;
modifying the first ML model;
accessing first event data describing a first event of the first event type;
executing the first ML model to determine an ML characterization of the first event using the first event data;
applying a first rule set to the first event data to generate a rule characterization of the first event;
determining an output characterization of the first event based at least in part on the rule characterization of the first event; and
determining to deactivate the first rule set based at least in part on the ML characterization of the first event.

12. The system of claim 11 , the operations further comprising, before applying the first rule set to the first event data, applying a proactive detection rule set to the first event data to determine that the first event is an untrained event.

13. The system of claim 11 , the operations further comprising:
accessing second event data describing a second event of a second event type different than the first event type;
executing the first ML model to determine an ML characterization of the second event using the second event data;
determining that a confidence of the ML characterization of the second event is less than a threshold level;
applying a reactive detection rule set to the first event data to determine that the second event is not an untrained event; and
responsive to determining that the second is not an untrained event, sending an indication of the second event to a manual processing queue.

14. The system of claim 11 , the operations further comprising:
accessing second event data describing a second event of a second event type different than the first event type;
executing the first ML model to determine an ML characterization of the second event using the second event data;
determining that a confidence of the ML characterization of the second event is less than a confidence threshold; and
applying a reactive detection rule set to the second event data to determine that the second event is an untrained event, the

Assignees

• Sap Se

Inventors

• Eberlein Peter
• Driesen Volker

Classifications

• G06N5/025

  Extracting rules from data · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06Q10/1053 Primary

  Employment or hiring · CPC title