Dynamic multi-model monitoring and validation for artificial intelligence models

Patent metadata

Field	Value
Publication number	US-12299140-B2
Application number	US-202418947102-A
Country	US
Kind code	B2
Filing date	Nov 14, 2024
Priority date	Jan 19, 2023
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

The systems and methods disclosed herein receives artifacts generated using a first set of models within a multi-model superstructure. The multi-model superstructure includes a second set of models to test the first set of models. The multi-model superstructure dynamically routes the artifacts of the first set of models to one or more models of the second set of models by (i) determining a set of dimensions of the artifacts against which to evaluate the artifacts and (ii) identifying the models in the second set used to test the particular dimension. The second set of models then assesses each artifact against a set of assessment metrics. If an artifact fails to meet one or more assessment metrics, the second set of models generates actions to align the artifact with the set of assessment metrics.

First claim

We claim:

1. A non-transitory computer-readable storage medium comprising instructions thereon, wherein the instructions when executed by at least one data processor of a system, cause the system to:
receive, from a computing device, an output generation request comprising a prompt for generation of an output using a multi-model superstructure, the multi-model superstructure comprising: (i) a first set of AI models and (ii) a second set of AI models;
supply the output generation request to one or more AI models of the first set of AI models to generate a set of model-specific outputs;
dynamically route, by the multi-model superstructure, the set of model-specific outputs of the first set of AI models to one or more AI models of the second set of AI models by:
determining a set of dimensions of the set of model-specific outputs against which to evaluate the set of model-specific outputs, and
for each particular dimension in the determined set of dimensions, identifying, by the multi-model superstructure, the one or more AI models of the second set of AI models used to test the particular dimension;

for each particular dimension in the determined set of dimensions, evaluate, by the second set of AI models, each particular model-specific output of the set of model-specific outputs against a set of assessments to determine satisfaction of the particular model-specific output with a corresponding set of assessment metrics of each assessment by:
constructing the set of assessments including a set of seed assessments testing the particular dimension of the particular model-specific output against threshold values of the corresponding set of assessment metrics,
comparing values of the corresponding set of assessment metrics of the particular model-specific output with the threshold values of the corresponding set of assessment metrics,
using the comparison, generating a set of seed assessment results indicating a degree of satisfaction of the particular model-specific output with the threshold values of the corresponding set of assessment metrics of the set of seed assessments,
using the set of seed assessment results, dynamically constructing a set of subsequent assessments within the set of assessments constructed subsequent to the set of seed assessments, and
applying the set of subsequent assessments of the set of assessments to the particular model-specific output to generate a set of overall assessment results based on a degree of satisfaction of the particular model-specific output with the threshold values of the assessment metrics of: (i) the set of seed assessments and (ii) the set of subsequent assessments;

responsive to the set of assessment results of a particular model-specific output failing to satisfy one or more threshold values of the corresponding set of assessment metrics of the set of assessments, generate, by the second set of AI models, a set of actions to add a set of pre-loaded query context to the output generation request indicated by the particular assessment metrics; and
for each model-specific output, cause the computing device to display a graphical layout indicating the set of assessment results,
wherein the graphical layout includes (1) a first representation of the particular model-specific output and (2) a second representation of a corresponding set of actions generated.

2. The non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:
using the generated set of actions, update the output generation request by automatically triggering an automated workflow indicated by the generated set of actions, wherein the automated workflow includes executing the generated set of actions;
using the updated output generation request, supply the updated output generation request to the one or more AI models of the first set of AI models to generate a set of updated model-specific outputs; and
evaluate, by the second set of AI models, each particular updated model-specific output of the updated model-specific outputs against the set of assessments to determine satisfaction of the particular updated model-specific output with the corresponding set of assessment metrics of each assessment.

3. The non-transitory, computer-readable storage medium of claim 1 , wherein the set of model-specific outputs is a first set of model-specific outputs, wherein the instructions further cause the system to:
provide the output generation request loaded with the pre-loaded query context to the one or more AI models of the first set of AI models to generate a second set of model-specific outputs; and
responsive to the second set of model-specific outputs satisfying each assessment metrics of the set of assessments, automatically transmit, to the computing device, the second set of model-specific outputs.

4. The non-transitory, computer-readable storage medium of claim 1 , wherein the one or more AI models of the second set of AI models are determined randomly.

5. The non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:
establish a predefined schedule to rotate the one or more AI models in the second set of AI models,
wherein the predefined schedule is established using one or more of: (1) time intervals or (2) a number of output generation requests processed; and

using the predefined schedule, determine the one or more AI models of the second set of AI models.

6. The non-transitory, computer-readable storage medium of claim 1 ,
wherein the first set of AI models and the second set of AI models include one or more of: 1) general-purpose AI models or 2) domain-specific AI models,
wherein the set of model-specific outputs are routed to the one or more AI models of the second set of AI models trained on data sharing a common domain with one or more model-specific outputs of the set of model-specific outputs, and
wherein the domain indicates one or more of: 1) an area of knowledge, 2) a data type, 3) a guideline type, or 4) a type of task.

7. The non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system to:
determine whether the particular model-specific output fails to satisfy one or more particular assessment metrics of the set of assessments using a majority vote between the one or more AI models of the second set of AI models.

8. A computing system for dynamic multi-model monitoring and validation of a generative artificial intelligence model, the system comprising:
a first set of AI models configured to generate a set of artifacts;
a second set of AI models configured to receive, from a computing device, the set of artifacts as input to generate, using at least one hardware processor of the computing device, a set of assessment results indicating a degree of satisfaction of a set of assessment metric values of the set of artifacts with a set of threshold metric values of a corresponding set of assessment metrics; and
a third set of AI models configured to:
dynamically route the set of artifacts generated by the first set of AI models to one or more AI models in the second set of AI models by:
determining a set of dimensions of the set of artifacts against which to evaluate the set of artifacts,
for each particular dimension in the determined set of dimensions, identifying the one or more AI models of the second set of AI models used to test the particular dimension, and

responsive to the set of assessment results generated by the second set of AI models failing to satisfy one or more threshold metric values of the corresponding set of assessment metrics, cause the second set of AI models to generate a set of actions to modify one or more of: 1) parameter

Assignees

• Citibank Na

Inventors

• Rahman Sofia
• Myers James
• Praveen Prashant
• Malviya Shardul
• Liao Wayne
• Jain Deepak
• Cory Samantha
• Saternus Mariusz
• Lewandowski Daniel
• Rath Biraj Krushna
• Murray Stuart
• Davies Philip
• Jain Payal
• Maonah Tariq Husayn
• Mysore Vishal
• Ayyadurai Ramkumar
• Desilva Chamindra
• Cameron William Franklin
• Silver Miriam
• Rao Prithvi Narayana
• Goyal Pramod
• Rajaretnam Manjit

Classifications

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06F21/552

  involving long-term monitoring or reporting · CPC title

• G06F21/577 Primary

  Assessing vulnerabilities and evaluating computer system security · CPC title