Identifying and remediating gaps in artificial intelligence use cases using a generative artificial intelligence model

Patent metadata

Field	Value
Publication number	US-12505352-B2
Application number	US-202519226048-A
Country	US
Kind code	B2
Filing date	Jun 2, 2025
Priority date	Apr 11, 2024
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

The systems and methods disclosed herein receive alphanumeric characters defining operative boundaries for expected model use cases, along with operational data. The expected model use cases share common attributes, which are used by a first AI model to construct observed model use cases from the operational data. Each observed model use case includes features such as a text-based description, expected input and output,? AI model(s) generating the expected output from the input, and/or data supporting the AI models. For each observed model use case, a second AI model maps the alphanumeric characters and features to a risk category, selecting from multiple risk categories based on the level of risk associated with the features. The system identifies criteria for the observed model use case within the alphanumeric characters and generates gaps by comparing the criteria with the features of the observed model use case.

First claim

We claim:

1 . One or more non-transitory computer-readable storage media comprising instructions thereon, wherein the instructions when executed by at least one data processor of a system, cause the system to:
obtain (1) an alphanumeric character set representing one or more operative boundaries of one or more reference model use cases and (2) an operational data set; determine an observed model use case set of the operational data set using the alphanumeric character set,
wherein the observed model use case set is associated with a feature set including one or more of: a text-based description, an expected input, an expected output, one or more Al models configured to generate the expected output using the expected input, or data supporting the one or more Al models;
link the feature set of the observed model use case set to a risk category indicated within the alphanumeric character set; determine a criteria set corresponding to the observed model use case set within the alphanumeric character set based on the linked risk category; generate a first gap set for the observed model use case set by comparing the criteria set corresponding to the observed model use case set with the feature set of the observed model use case set; present a representation that includes one or more of: a user interface component or a set of text via a user interface of a computing device, wherein the representation indicates at least one of: (1) the first gap set or (2) a metadata set linked to the first gap set; responsive to obtaining a subsequent input via the user interface of the computing device, update the representation by modifying the feature set of the observed model use case set; and generate a second gap set for the observed model use case set to validate satisfaction of the modified feature set of the observed model use case set with the criteria set.

2 . The one or more non-transitory computer-readable storage media of claim 1 , wherein the metadata set includes a timestamp of determining the observed model use case set.

3 . The one or more non-transitory computer-readable storage media of claim 2 , wherein the instructions further cause the system to:
generate a document set associated with the observed model use case set,
wherein the document set includes the timestamp.

4 . The one or more non-transitory computer-readable storage media of claim 3 , wherein the instructions further cause the system to:
using the first gap set, generate an action set configured to cause the feature set of the observed model use case set to satisfy the criteria set,
wherein updating the representation is in accordance with the action set.

5 . The one or more non-transitory computer-readable storage media of claim 4 , wherein the instructions further cause the system to:
using the generated action set, automatically trigger an automated workflow indicated by the generated action set,
wherein the automated workflow includes executing the generated action set.

6 . The one or more non-transitory computer-readable storage media of claim 1 , wherein the instructions further cause the system to:
monitor satisfaction of the feature set of the observed model use case set with the criteria set of the observed model use case set according to an interval set; and subsequent to determining that the feature set of the observed model use case set fails to satisfy the criteria set, update the first gap set.

7 . The one or more non-transitory computer-readable storage media of claim 1 , wherein the instructions further cause the system to:
determine, for each gap in the first gap set, a gap type between the criteria set of the observed model use case set and the feature set of the observed model use case set; and trigger one or more alarms in response to the gap type of a particular gap satisfying a threshold.

8 . A system comprising:
at least one hardware processor; and at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
obtain (1) a guideline set representing one or more operative boundaries of one or more reference model use cases and (2) an operational data set;
determine an observed model use case set of the operational data set using the guideline set,
wherein the observed model use case set is associated with a feature set including one or more of: a text-based description, an expected input, an expected output, one or more AI models configured to generate the expected output using the expected input, or data supporting the one or more AI models;

associate the feature set of the observed model use case set with a risk category indicated within the guideline set;
determine a criteria set corresponding to the observed model use case set within the guideline set based on the associated risk category;
generate a first gap set for the observed model use case set by comparing the criteria set corresponding to the observed model use case set with the feature set of the observed model use case set;
transmit, via a computing device, a representation for the observed model use case set that indicates at least one of: (1) the first gap set or (2) a metadata set associated with the first gap set;
responsive to obtaining a subsequent input via the computing device, modifying the feature set of the observed model use case set; and
update the first gap set for the observed model use case set to validate satisfaction of the modified feature set of the observed model use case set with the criteria set.

9 . The system of claim 8 , wherein the system is further caused to:
compare an expected attribute set indicated by the criteria set with an observed attribute set defining the feature set of the observed model use case set; and subsequent to determining that one or more attributes in the expected attribute set are absent from the observed attribute set, generate an action set configured to add the one or more attributes in the expected attribute set absent from the observed attribute set to the observed attribute set.

10 . The system of claim 8 , wherein the system is further caused to:
transmit a request for an additional feature set associated with the observed model use case set; subsequent to obtaining the additional feature set, modify the feature set of the observed model use case set; and determine that the modified feature set of the observed model use case set satisfies the criteria set.

11 . The system of claim 8 , wherein the metadata set indicates one or more of: a timestamp or a version.

12 . The system of claim 8 , wherein the system is further caused to:
transmit (1) the criteria set and (2) the observed model use case set into one or more nodes of an input layer of an AI model to generate the first gap set.

13 . The system of claim 8 , wherein the system is further caused to:
generate a document set associated with the observed model use case set, wherein the document set is linked to a timestamp.

14 . A computer-implemented method, the computer-implemented method comprising:
obtaining (1) a guideline set representing one or more operative boundaries of one or more reference model use cases and (2) an operational data set; determining an observed model use case set of the operational data set using the guideline set, transmitting each particular observed model use case of the observed model use case set into one or more nodes of an input layer of an AI model set trained to:
link a feature set of the observed model use case set to a risk category indicated within the guideline set, and
generate a

Assignees

• Citibank Na

Inventors

• Rahman Sofia
• Tucker Christopher
• Myers James
• Praveen Prashant
• Malviya Shardul
• Liao Wayne
• Jain Deepak
• Cory Samantha
• Saternus Mariusz
• Lewandowski Daniel
• Rath Biraj Krushna
• Murray Stuart
• Davies Philip
• Jain Payal
• Maonah Tariq Husayn
• Mysore Vishal
• Ayyadurai Ramkumar
• Desilva Chamindra

Classifications

• G06F21/6218

  to a system of files or objects, e.g. local or distributed file system or database · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F21/31

  User authentication · CPC title