Below-the-line thresholds tuning with machine learning

Patent metadata

Field	Value
Publication number	US-2020104849-A1
Application number	US-201816145952-A
Country	US
Kind code	A1
Filing date	Sep 28, 2018
Priority date	Sep 28, 2018
Publication date	Apr 2, 2020
Grant date	—

Abstract

Systems, methods, and other embodiments associated with applying machine learning to below-the-line threshold tuning are described. In one embodiment, a method includes selecting a set of sampled events and labeling each event in the set of sampled events as either suspicious or not suspicious. Then, a machine learning model to calculate for a given event a probability that the given event is suspicious is built based on the set of sampled events. The machine learning model is trained, and its calibration validated. Based on probabilities calculated by the machine learning model, a scenario and segment combination to be tuned is determined. A tuned threshold value is generated, and an alerting engine is adjusted with the tuned parameter to reduce errors by the alerting engine in classifying events as not suspicious.

First claim

What is claimed is:

1 . A non-transitory computer-readable medium storing computer-executable instructions that when executed by at least a processor of a computer cause the computer to:
select, by at least the processor, a set of sampled events from a set of historic events previously divided by an alerting engine into a set of below-the-line events and a set of above-the-line events separated by a threshold line indicating that an event is suspicious, wherein the threshold line is defined at least in part by one or more threshold values; label, by at least the processor, each event in the set of sampled events as either suspicious or not suspicious; build, by at least the processor, based at least in part on the set of sampled events, a machine learning model to calculate for a given event a probability that the given event is suspicious; train, by at least the processor, the machine learning model; validate, by at least the processor, that the machine learning model is calibrated; determine, by at least the processor, based at least in part on one or more probabilities calculated by the machine learning model, a scenario and segment combination to be tuned; generate, by at least the processor, a tuned threshold value based at least in part on the one or more probabilities calculated by the machine learning model; and tune, by at least the processor, the alerting engine by replacing at least one of the one or more threshold values with the tuned threshold value to cause the threshold line to be adjusted to reduce errors by the alerting engine in classifying events as not suspicious.

2 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that when executed by at least the processor cause the processor to:
select, by at least the processor, a preliminary set of sampled events from the set of below-the-line events; and determine, by at least the processor, whether any event in the preliminary set is suspicious.

3 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that when executed by at least the processor cause the processor to:
pool, by at least the processor, together two or more events in the set of historic events by scenario and segment combination applied to a focal entity, thereby creating a set of pooled events; and correlate, by at least the processor, each event in the set of pooled events across all scenario and segment combinations by focal entity, thereby creating a set of correlated events.

4 . The non-transitory computer-readable medium of claim 1 , wherein the instructions that cause the processor to select further comprise instructions that when executed by at least the processor cause the processor to deduplicate the set of sampled events such that each correlated event is unique based on a primary key.

5 . The non-transitory computer-readable medium of claim 3 , further comprising instructions that when executed by at least the processor cause the processor to:
revert, by at least the processor, the set of correlated events to the set of pooled events following a determination that a first estimated probability for the set of correlated events exceeds a given value; correlate, by at least the processor, each event in the set of pooled events within each scenario and segment combination by focal entity, thereby creating a set of combination correlated events; and identify, by at least the processor, a one of the set of combination correlated event sets where a second estimated probability for the one of the set exceeds the given value; wherein (i) the generation of the tuned threshold value is further initiated based at least in part on the second estimated probability, and (ii) the at least one of the one or more threshold values is associated with the scenario and segment combination of the one of the set.

6 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that when executed by at least the processor cause the processor to repeat the selection and labeling by at least the processor at least until a predetermined number of events have been labeled suspicious, the predetermined number being an amount of suspicious events sufficient to build and validate the machine learning model.

7 . The non-transitory computer-readable medium of claim 1 , further comprising instructions that when executed by at least the processor cause the processor to:
select, by at least the processor, a training set of events from the set of sampled events; determine, by at least the processor, using the machine learning model, a probability that the event is suspicious for each event in the training set; sort, by at least the processor, the events of the training set in order of probability that the event is suspicious; divide, by at least the processor, the sorted training set into two or more sections, each section having an approximately equal number of events, an upper boundary probability that an event is suspicious and a lower boundary probability that an event is suspicious; and assign, by at least the processor, to each section, an expected probability that an event is suspicious.

8 . The non-transitory computer-readable medium of claim 7 , wherein the instructions that when executed by at least the processor cause the processor to validate that the machine learning model is calibrated further comprise:
select, by at least the processor, a validation set of events from the set of sampled events; determine, by at least the processor, using the machine learning model, a probability that the event is suspicious for each event in the validation set; sort, by at least the processor, the events of the validation set in order of probability that the event is suspicious; divide, by at least the processor, the sorted validation set into sections corresponding with the sections of the training set, wherein there is one section of the validation set for each section of the training set, and each section of the validation set has the same an upper boundary probability that an event is suspicious and a lower boundary probability that an event is suspicious as does the corresponding section of the training set; and determine, by at least the processor, that the machine learning model is calibrated if, for each section of the validation set, the expected probability of the corresponding section of the training set does not underestimate an observed rate of suspicious events in the section of the validation set.

9 . A computer-implemented method, comprising:
selecting a set of sampled events from a set of historic events previously divided by an alerting engine into a set of below-the-line events and a set of above-the-line events separated by a threshold line indicating that an event is suspicious, wherein the threshold line is defined at least in part by one or more threshold values; labeling each event in the set of sampled events as either suspicious or not suspicious; building, based at least in part on the set of sampled events, a machine learning model to calculate for a given event a probability that the given event is suspicious; training the machine learning model; validating that the machine learning model is calibrated; determining, based at least in part on one or more probabilities calculated by the machine learning model, a scenario and segment combination to be tuned; generating a tuned threshold value based at least in part on the one or more probabilities calculated by the machine learning model; and tuning the alerting engine by replacing at least one of the one or more threshold values with the tuned threshold value to cause the threshold line to be adjusted t

Assignees

• Oracle Financial Services Software Ltd

Inventors

• Cai Jian
• Mathew Sunil J

Classifications

• G06Q40/00

  Finance; Insurance; Tax strategies; Processing of corporate or income taxes · CPC title

• G06Q20/4016 Primary

  involving fraud or risk level assessment in transaction processing · CPC title

• G06Q40/02

  Banking, e.g. interest calculation or account maintenance (credit or loans G06Q40/03) · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F15/18

  Physics · mapped topic