Risk-aware model ranking

Patent metadata

Field	Value
Publication number	US-2025390637-A1
Application number	US-202418753001-A
Country	US
Kind code	A1
Filing date	Jun 25, 2024
Priority date	Jun 25, 2024
Publication date	Dec 25, 2025
Grant date	—

Abstract

Embodiments of the invention provide a computer-implemented method that includes executing, using a processor system, a risk-aware model evaluation. Executing the risk-aware model evaluation includes the risk-aware model evaluation analyzing a dataset that represents one or more non-risk-aware performance metrics associated with one or more models-under-evaluation. Analyzing the dataset comprises determining a subset of the dataset based at least in part on a determination that the subset satisfies one or more risk severity criteria.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
executing, using a processor system, a risk-aware model evaluation; wherein executing the risk-aware model evaluation comprises the risk-aware model evaluation analyzing a dataset that represents one or more non-risk-aware performance metrics associated with one or more models-under-evaluation; and wherein analyzing the dataset comprises determining a subset of the dataset based at least in part on a determination that the subset satisfies one or more risk severity criteria.

2 . The computer-implemented method of claim 1 , wherein executing the risk-aware model evaluation further comprises generating a risk severity score based at least in part on the subset.

3 . The computer-implemented method of claim 2 , wherein:
the one or more models-under-evaluation comprise a first model-under-evaluation and a second model-under-evaluation; and generating the risk severity score based at least in part on the subset comprises generating a first risk severity score associated with the first model-under-evaluation and generating a second risk severity score associated with the second model-under-evaluation.

4 . The computer-implemented method of claim 3 , wherein generating the risk severity score based at least in part on the subset further comprises comparing the first risk severity score associated with the first model-under-evaluation to the second risk severity score associated with the second model-under-evaluation using a stochastic dominance evaluation.

5 . The computer-implemented method of claim 4 , wherein generating the risk severity score based at least in part on the subset further comprises further evaluating a relationship between the first risk severity score associated with the first model-under-evaluation and the second risk severity score associated with the second model-under-evaluation using an optimal transportation analysis.

6 . The computer-implemented method of claim 1 , wherein:
the dataset comprises a first dataset and a second dataset; the one or more models-under-evaluation comprises a first model-under-evaluation and a second model-under-evaluation; the first dataset is associated with the first model-under-evaluation; the second dataset is associated with the second model-under-evaluation; the subset comprises a first subset associated with the first dataset and a second subset associated with the second dataset; and the one or more non-risk-aware performance metrics comprises a first non-risk-aware performance metric.

7 . The computer-implemented method of claim 6 , wherein the determination that the subset satisfies the one or more risk severity criteria is based at least in part on:
a determination that the first subset satisfies the one or more risk severity criteria; and a determination that the second subset satisfies the one or more risk severity criteria.

8 . The computer-implemented method of claim 7 , wherein:
executing the risk-aware model evaluation further comprises generating a risk severity score based at least in part on the subset; generating the risk severity score based at least in part on the subset further comprises:
generating a first risk severity score associated with the first model-under-evaluation and the first non-risk-aware performance metric;
generating a second risk severity score associated with the second model-under-evaluation and the first non-risk-aware performance metric;
comparing the first risk severity score associated with the first model-under-evaluation to the second risk severity score associated with the second model-under-evaluation using a stochastic dominance evaluation; and
evaluating a relationship between the first risk severity score associated with the first model-under-evaluation and the second risk severity score associated with the second model-under-evaluation using an optimal transportation analysis.

9 . A computer system comprising a processor system and a memory electronically coupled to the processor system, wherein the processor system is operable to perform processor system operations comprising:
executing a risk-aware model evaluation; wherein executing the risk-aware model evaluation comprises the risk-aware model evaluation analyzing a dataset that represents one or more non-risk-aware performance metrics associated with one or more models-under-evaluation; and wherein analyzing the dataset comprises determining a subset of the dataset based at least in part on a determination that the subset satisfies one or more risk severity criteria.

10 . The computer system of claim 9 , wherein executing the risk-aware model evaluation further comprises generating a risk severity score based at least in part on the subset.

11 . The computer system of claim 10 , wherein:
the one or more models-under-evaluation comprise a first model-under-evaluation and a second model-under-evaluation; and generating the risk severity score based at least in part on the subset comprises generating a first risk severity score associated with the first model-under-evaluation and generating a second risk severity score associated with the second model-under-evaluation.

12 . The computer system of claim 11 , wherein generating the risk severity score based at least in part on the subset further comprises comparing the first risk severity score associated with the first model-under-evaluation to the second risk severity score associated with the second model-under-evaluation using a stochastic dominance evaluation.

13 . The computer system of claim 12 , wherein generating the risk severity score based at least in part on the subset further comprises further evaluating a relationship between the first risk severity score associated with the first model-under-evaluation and the second risk severity score associated with the second model-under-evaluation using an optimal transportation analysis.

14 . The computer system of claim 9 , wherein:
the dataset comprises a first dataset and a second dataset; the one or more models-under-evaluation comprises a first model-under-evaluation and a second model-under-evaluation; the first dataset is associated with the first model-under-evaluation; the second dataset is associated with the second model-under-evaluation; the subset comprises a first subset associated with the first dataset and a second subset associated with the second dataset; and the one or more non-risk-aware performance metrics comprises a first non-risk-aware performance metric.

15 . The computer system of claim 14 , wherein the determination that the subset satisfies the one or more risk severity criteria is based at least in part on:
a determination that the first subset satisfies the one or more risk severity criteria; and a determination that the second subset satisfies the one or more risk severity criteria.

16 . The computer system of claim 15 , wherein:
executing the risk-aware model evaluation further comprises generating a risk severity score based at least in part on the subset; generating the risk severity score based at least in part on the subset further comprises:
generating a first risk severity score associated with the first model-under-evaluation and the first non-risk-aware performance metric;
generating a second risk severity score associated with the second model-under-evaluation and the first non-risk-aware performance metric;
comparing the first risk severity score associated with the first model-under-evaluation to the second risk severity score associated with the second

Assignees

• IBM

Inventors

• Nitsure Apoorva
• Mroueh Youssef
• Rigotti Mattia
• Greenewald Kristjan Herbert
• Belgodere Brian Michael
• Yurochkin Mikhail
• Navratil Jiri
• Melnyk Igor
• Ross Jarret

Classifications

• G06F30/27 Primary

  using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model · CPC title