Automatic Alert Dispositioning using Artificial Intelligence

Patent metadata

Field	Value
Publication number	US-2024177054-A1
Application number	US-202218072153-A
Country	US
Kind code	A1
Filing date	Nov 30, 2022
Priority date	Nov 30, 2022
Publication date	May 30, 2024
Grant date	—

Abstract

The system identifies false positives about a user and closes resulting alerts before the alert wastefully consume system resources. Machine learning techniques including clustering and multi-labeling classification are used to effectively categorize prior text-based notes to efficiently identify and automatically close false positive alerts. Moreover, weak labeling, AI transformers/sentence transformation, and/or k-means cluster analysis provide a means for condensing large quantities of textual data into ML model components with improved interpretability. Customer relationship management (CRM) platform and Risk Management Supervision (RMS) note analysis captures inefficiently/ineffectively organized past work and leverages it to reduce redundancies in future expert user/supervisory/customer advisory efforts. The various ML techniques disclosed herein output numerical features that directly improve model performance for alert triaging as a whole. Streamlining the presentation of large textual data to reviewers/users (e.g., supervision principals) as individual sentences or numerical values greatly improves explainability as a byproduct.

First claim

We claim:

1 . A method for automated triaging of alerts generated for customer service advisors to reduce false positive alerts using artificial intelligence (AI) sentence transformers, weak labeling in machine learning, and multi-label classification of textual notes about interactions between the customer service advisors and customers, the method comprising:
receiving as input, from a customer relationship management (CRM) platform, CRM notes that comprise text related to a customer; receiving a plurality of user-defined questions relating to each of an enumerated list of scenarios; generating, using an AI sentence transformer, a matrix with CRM note-question pairs that captures relevance of each CRM note to a first scenario in the enumerated list of scenarios, wherein the pairs are structured as input embeddings; aggregating, using a weak labeling system, results by applying user-defined labeling functions to the CRM notes, wherein the aggregated results provide a holistic weak label for each CRM note; training a discriminative model using heuristics labels generated by the weak labeling system after the aggregating step; updating, based on an output of the AI sentence transformer and the discriminative model, a machine learning (ML) model, which was trained based on k-means clustering outputs from a text-frequency matrix created from RMS notes; and suppressing, by the ML model, an alert about the customer based on a likelihood that the alert corresponding to the customer is false positive, wherein the alert is for a customer with a likelihood of satisfying the first scenario from among the enumerated list of scenarios, wherein unsuppressed alerts are displayed on a graphical user interface (GUI) to a customer service advisor responsible for servicing the customer.

2 . The method of claim 1 , wherein the ML model:
infers a likelihood that the customer was previously notified of the first scenario by clustering numerical representations of the RMS notes, which represent interactions between the customer service advisor and the customer, in relation to the enumerated list of scenarios; and outputs a multi-classification relevance score associated with the customer, wherein the multi-classification relevance score comprises a relevance score corresponding to the first scenario, and wherein each relevance score in the multi-classification relevance score is calculated as a distance from a corresponding numerical representation to a centroid of a corresponding cluster.

3 . The method of claim 1 , wherein the ML model was created by steps comprising:
filtering keywords in RMS notes to remove a first set of keywords that fail to correspond to any specific scenario among the enumerated list of scenarios; and after the filtering of the keywords, transforming the plurality of RMS notes into a text-frequency matrix, wherein the first set of keywords are omitted from the text-frequency matrix, and wherein each row of the text-frequency matrix represents a different RMS note.

4 . The method of claim 3 , wherein the ML model was created by steps comprising:
reducing a dimension of the text-frequency matrix, wherein the text-frequency matrix is outputted as numerical representations; k-means clustering on the numerical representations outputted by the text-frequency matrix to produce a plurality of clusters, each cluster corresponding to a single scenario in the enumerated list of scenarios; and for each RMS note, outputting a multi-classification relevance score that includes a quantity of scores matching a quantity of clusters in the plurality of clusters, wherein each score in the multi-classification relevance score is calculated as a distance from the corresponding numerical representation to a centroid of a corresponding cluster, wherein the multi-classification relevance score corresponds to a likelihood that a particular RMS note belongs to a specific scenario.

5 . The method of claim 1 , further comprising: tagging a CRM note as corresponding to the first scenario of the enumerated list of scenarios based on the output of the AI sentence transformer.

6 . The method of claim 1 , wherein the user-defined labeling functions comprise labeling based on length of a note and presence of particular keywords.

7 . The method of claim 1 , wherein the aggregating is performed by calculating an unweighted average.

8 . The method of claim 1 , further comprising: outputting an accuracy of a labeling function of the user-defined labeling functions.

9 . The method of claim 1 , wherein the customer service advisor includes an AI-chat bot.

10 . The method of claim 1 , wherein the plurality of user-defined questions used with the AI sentence transformer correspond to features for feature engineering of the ML model that predicts the likelihood that the alert corresponding to the customer is false positive.

11 . The method of claim 1 , wherein the aggregating of the results includes a majority vote of the user-defined labeling functions.

12 . The method of claim 1 , wherein the likelihood that the alert corresponding to the customer is false positive further comprises: the ML model taking as inputs prior occurrences of alerts, total assets of the customer, recent losses of the customer, types of securities traded by the customer, and birthdate of the customer.

13 . A non-transitory computer-readable medium storing computer-executable instruction, which when executed by a computer processor, cause the computer processor to perform steps comprising:
receiving as input, from a customer relationship management (CRM) platform, CRM notes that comprise text related to a customer; receiving a plurality of user-defined questions relating to each of an enumerated list of scenarios; generating, using an AI sentence transformer, a matrix with CRM note-question pairs that captures relevance of each CRM note to a first scenario in the enumerated list of scenarios, wherein the pairs are structured as input embeddings; aggregating, using a weak labeling system, results by applying user-defined labeling functions to the CRM notes, wherein the aggregated results provide a holistic weak label for each CRM note; training a discriminative model using heuristics labels generated by the weak labeling system after the aggregating step; updating, based on an output of the AI sentence transformer and the discriminative model, a machine learning (ML) model, which was trained based on k-means soft clustering outputs from a text-frequency matrix created from RMS notes; and suppressing, by the ML model, an alert about the customer based on a likelihood that the alert corresponding to the customer is false positive, wherein the alert is for a customer with a likelihood of satisfying the first scenario from among the enumerated list of scenarios, wherein unsuppressed alerts are displayed on a graphical user interface (GUI) to a customer service advisor responsible for servicing the customer.

14 . The non-transitory computer-readable medium of claim 13 , wherein the ML model:
infers a likelihood that the customer was previously notified of the first scenario by clustering numerical representations of the RMS notes, which represent interactions between the customer service advisor and the customer, in relation to the enumerated list of scenarios; and outputs a multi-classification relevance score associated with the customer, wherein the multi-classification relevance score comprises a relevance score corresponding to the first scenario, and wherein each relevance score in the multi-classification relevance score is calculated as a distance fr

Assignees

• Bank Of America

Inventors

• Li Yvonne
• Chan Franklin Kaiyuen
• Kim Min Kyung
• Ginsberg Jared Scott
• Blogg Aaron

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• H04L51/02

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• G06Q40/06

  Asset management; Financial planning or analysis · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• H04L63/1425

  Traffic logging, e.g. anomaly detection · CPC title