Systems and methods for accelered detection and replacement of anomalous machine learning-based digital threat scoring ensembles and intelligent generation of anomalous artifacts for anomalous ensembles

Patent metadata

Field	Value
Publication number	US-2023124621-A1
Application number	US-202217963365-A
Country	US
Kind code	A1
Filing date	Oct 11, 2022
Priority date	Oct 11, 2021
Publication date	Apr 20, 2023
Grant date	—

Abstract

A system and method for accelerated anomaly detection and replacement of an anomaly-experiencing machine learning-based ensemble includes identifying a machine learning-based digital threat scoring ensemble having an anomalous drift behavior in digital threat score inferences computed by the machine learning-based digital threat scoring ensemble for a target period; executing a tiered anomaly evaluation for the machine learning-based digital threat scoring ensemble that includes identifying at least one errant machine learning-based model of the machine learning-based digital threat scoring ensemble contributing to the anomalous drift behavior, and identifying at least one errant feature variable of the at least one machine learning-based model contributing to the anomalous drift behavior; generating a successor machine learning-based digital threat scoring ensemble to the machine learning-based digital threat scoring ensemble based on the tiered anomaly evaluation; and replacing the machine learning-based digital threat scoring ensemble with the successor machine learning-based digital threat scoring ensemble.

First claim

We claim:

1 . A method for accelerated anomaly detection and replacement of an anomaly-experiencing machine learning-based ensemble, the method comprising:
identifying, by one or more computers, a machine learning-based digital threat scoring ensemble having an anomalous drift behavior in digital threat score inferences computed by the machine learning-based digital threat scoring ensemble for a target period; executing, based on the identifying, a tiered anomaly evaluation for the machine learning-based digital threat scoring ensemble, wherein the tiered anomaly evaluation includes:
(a) identifying at least one machine learning-based model of the machine learning-based digital threat scoring ensemble contributing to the anomalous drift behavior, and
(b) identifying at least one feature variable of the at least one machine learning-based model contributing to the anomalous drift behavior;
generating a potential successor machine learning-based digital threat scoring ensemble to the machine learning-based digital threat scoring ensemble based on the tiered anomaly evaluation, wherein the potential successor machine learning-based digital threat scoring ensemble mitigates the anomalous drift behavior; and replacing the machine learning-based digital threat scoring ensemble with the potential successor machine learning-based digital threat scoring ensemble based on one or more ensemble metrics computed for the potential successor machine learning-based digital threat scoring ensemble satisfying one or more efficacy benchmarks.

2 . The method according to claim 1 , further comprising:
sourcing, by the one or more computers, threat score distribution data computed by the machine learning-based digital threat scoring ensemble for each day included within the target period; and detecting, by the one or more computers, an occurrence of a statistically significant upward trend or a statistically significant downward trend in the threat score distribution data over the target period, wherein:
the identifying the machine learning-based digital threat scoring ensemble is based on the detecting of the statistically significant upward trend or the statistically significant downward trend.

3 . The method according to claim 1 , further comprising:
executing one or more intelligent ensemble simulations that inform a structure of the potential successor machine learning-based digital threat scoring ensemble, and the structure of the potential successor machine learning-based digital threat scoring ensemble excludes the at least one machine learning-based model of the machine learning-based digital threat scoring ensemble contributing to the anomalous drift behavior, wherein the generating the potential successor machine learning-based digital threat scoring ensemble is further based on the execution of the one or more intelligent ensemble simulations.

4 . The method according to claim 1 , further comprising:
executing one or more intelligent ensemble simulations that inform a structure of the potential successor machine learning-based digital threat scoring ensemble, and the structure of the potential successor machine learning-based digital threat scoring ensemble includes a machine learning-based model of a distinct type in substitution of the at least one machine learning-based model of the machine learning-based digital threat scoring ensemble contributing to the anomalous drift behavior, wherein the generating the potential successor machine learning-based digital threat scoring ensemble is further based on the execution of the one or more intelligent ensemble simulations.

5 . The method according to claim 1 , further comprising:
executing one or more intelligent ensemble simulations that inform a structure of the potential successor machine learning-based digital threat scoring ensemble, and the structure of the potential successor machine learning-based digital threat scoring ensemble excludes, as model input, the at least one feature variable contributing to the anomalous drift behavior, wherein the generating the potential successor machine learning-based digital threat scoring ensemble is further based on the execution of the one or more intelligent ensemble simulations.

6 . The method according to claim 1 , further comprising:
executing one or more intelligent ensemble simulations that inform a structure of the potential successor machine learning-based digital threat scoring ensemble, and the structure of the potential successor machine learning-based digital threat scoring ensemble is associated with an increase in the learned feature weighting of the at least one feature variable contributing to the anomalous drift behavior.

7 . The method according to claim 1 , further comprising:
executing one or more intelligent ensemble simulations that inform a structure of the potential successor machine learning-based digital threat scoring ensemble, and the structure of the potential successor machine learning-based digital threat scoring ensemble is associated with a decrease in the learned feature weighting of the at least one feature variable contributing to the anomalous drift behavior.

8 . The method according to claim 1 , wherein
the machine learning-based digital threat scoring ensemble is one of a volume of distinct machine learning-based digital threat scoring ensembles in operational use by a digital threat mitigation service, and one or more machine learning-based digital threat scoring ensembles of the volume is implemented for a distinct subscriber subscribing to the digital threat mitigation service.

9 . The method according to claim 8 , further comprising:
generating an anomaly rationale based on findings data derived from the tiered anomaly evaluation for the at least one machine learning-based model or the at least one feature variable contributing to the anomalous drift behavior; displaying, on a web-based user interface of the digital threat mitigation service, one or more anomalous ensemble artifacts, wherein the one or more anomalous ensemble artifacts include: one or more pieces of explainable content that provides the anomaly rationale for the at least one machine learning-based model or the at least one feature variable contributing to the anomalous drift behavior.

10 . A method for accelerated drift detection and replacement of a drift-experiencing machine learning-based ensemble, the method comprising:
evaluating, by one or more computers, a plurality of machine learning-based digital threat scoring ensembles in operational use by a digital threat mitigation platform; identifying, by the one or more computers, an anomalous machine learning-based digital threat scoring ensemble experiencing an anomalous drift behavior based on the evaluation; executing, by the one or more computers, an anomaly evaluation for the anomalous machine learning-based digital threat scoring ensemble based on the identifying, wherein the anomaly evaluation includes:
detecting at least one errant machine learning-based model of the anomalous machine learning-based digital threat scoring ensemble contributing to the anomalous drift behavior;
generating, based on executing one or more intelligent ensemble simulations, a plurality of candidate successor machine learning-based ensembles to the anomalous machine learning-based digital threat scoring ensemble; and replacing, by the one or more computers, the anomalous machine learning-based digital threat scoring ensemble with one of the plurality of candidate successor machine learning-based ensembles based on one or more ensemble metrics computed for the one of the plurality of candidate successor machine learning-based

Assignees

• Sift Science Inc

Inventors

• Umesh Pradhan Bagur
• Zhuang Yuan
• Wang Hui
• Benavides Nicholas
• Liu Chang
• Bao Yanqing
• Liu Wei

Classifications

• G06F2221/034

  Test or assess a computer or a system · CPC title

• G06F21/577 Primary

  Assessing vulnerabilities and evaluating computer system security · CPC title

• G06F21/552

  involving long-term monitoring or reporting · CPC title

• G06F21/554

  involving event detection and direct action · CPC title

• G06N20/20

  Ensemble learning · CPC title