Systems and methods for a digital register of models monitoring changes in accuracy of artificial intelligence models

Patent metadata

Field	Value
Publication number	US-12513010-B2
Application number	US-202217930035-A
Country	US
Kind code	B2
Filing date	Sep 6, 2022
Priority date	Sep 6, 2022
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

Methods and systems described herein relate to the creation of a digital repository of artificial intelligence models that allows users to determine their individual fairness metric. More specifically, the methods and systems provide this digital repository by storing it on a blockchain network and tracking any changes made to the model and/or its fairness metric.

First claim

What is claimed is:

1 . A system for monitoring changes in accuracy of artificial intelligence models by storing an artificial intelligence model in a blockchain network, the system comprising:
one or more processors; and a non-transitory computer readable medium having instructions recorded thereon that when executed by the one or more processors causes operations comprising:
receiving a first request to determine an accuracy metric for a first artificial intelligence model, wherein the accuracy metric indicates whether an error rate for the first artificial intelligence model is consistent across all inputs corresponding to individuals in a dataset, wherein individuals refer to categories of mixed attributes in the dataset, and wherein the error rate is based on a number of false positives and negatives that the artificial intelligence model makes;
in response to receiving the first request, accessing an on-chain digital repository of artificial intelligence model attributes on a blockchain network, wherein the artificial intelligence model attributes comprise respective attribute sets for a plurality of artificial intelligence models; and
determining a first block comprising a first attribute set for the first artificial intelligence model, wherein the first block comprises first data generated at a first time, wherein the first attribute set comprises:
a first temporal network identifier corresponding to a first run time of the first artificial intelligence model;
a first artificial intelligence model network identifier corresponding to the first artificial intelligence model;
a first output network identifier identifying a first given output for a first given input to the first artificial intelligence model;
a first input network identifier for the first artificial intelligence model, wherein the first input network identifier corresponds to the first given input;

determining a second block comprising a second attribute set for the first artificial intelligence model, wherein the second block comprises second data generated at a second time, wherein the second attribute set comprises:
a second temporal network identifier corresponding to a second run time of the first artificial intelligence model;
the first artificial intelligence model network identifier corresponding to the first artificial intelligence model;
a second output network identifier identifying a second given output for a second given input to the first artificial intelligence model;
a second input network identifier for the first artificial intelligence model, wherein the second input network identifier corresponds to the second given input;

based on information stored in the first output network identifier and the second output network identifier, determining whether the first artificial intelligence model has increased or decreased its fairness by comparing the first output network identifier and the second output network identifier to determine a difference in accuracy metrics; and
generating for display, on a user interface, a recommendation based on the difference.

2 . A method for monitoring changes in accuracy of artificial intelligence models, the method comprising:
receiving a first request to determine an accuracy metric for a first artificial intelligence model; in response to receiving the first request, accessing a repository of model attributes on a blockchain network; and determining a first block comprising a first attribute set for the first artificial intelligence model, wherein the first block comprises first data generated at a first time, wherein the first attribute set comprises:
a first temporal network identifier corresponding to a first run time of the first artificial intelligence model;
a first artificial intelligence model network identifier corresponding to the first artificial intelligence model;
a first output network identifier identifying a first given output for a first given input to the first artificial intelligence model; and
determining a second block comprising a second attribute set for the first artificial intelligence model, wherein the second block comprises second data generated at a second time, wherein the second attribute set comprises:
a second temporal network identifier corresponding to a second run time of the first artificial intelligence model;
the first artificial intelligence model network identifier corresponding to the first artificial intelligence model;
a second output network identifier identifying a second given output for a second given input to the first artificial intelligence model;
comparing the first output network identifier to the second output network identifier to determine a difference in accuracy metrics; and generating for display, on a user interface, a recommendation based on the difference.

3 . The method of claim 2 , wherein the first attribute set includes a first input network identifier for the first artificial intelligence model, wherein the first input network identifier corresponds to the first given input.

4 . The method of claim 2 , further comprising:
accessing an off-chain digital repository of the artificial intelligence model attributes; retrieving an off-chain version of the first artificial intelligence model from the off-chain digital repository; processing an off-chain version of the first given input in the off-chain version of the first artificial intelligence model to generate an off-chain version of the first given output; generating a first hash value of the off-chain version of the first given output; and comparing the first hash value to a second hash value, wherein the second hash value is based on the first output network identifier.

5 . The method of claim 2 , further comprising:
determining a random nonce; and using the random nonce to store the first given input inside hashed data groups, wherein the first given input is saved within hashes of data with the random nonce.

6 . The method of claim 2 , wherein determining the accuracy metric comprises:
receiving the first given output for the first given input, wherein the first given input is included in a first dataset; determining an error rate for given output; determining an average error rate for outputs of the first artificial intelligence model for a subset of inputs from the first dataset; and comparing the average error rate to the error rate to determine the accuracy metric.

7 . The method of claim 2 , wherein determining the difference in accuracy metrics comprises:
determining a first accuracy metric based on the first output network identifier; determining a second accuracy metric based on the second output network identifier; and comparing the first accuracy metric to the second accuracy metric to determine the difference.

8 . The method of claim 2 , further comprising:
determining the difference in accuracy metrics; determining whether the difference is positive or negative; and selecting the recommendation based on whether the difference is positive or negative.

9 . The method of claim 8 , further comprising:
retrieving a user profile corresponding to the first artificial intelligence model; determining a blockchain network address corresponding to the user profile; and executing a blockchain network function corresponding to the blockchain network address based on the recommendation.

10 . The method of claim 2 , wherein accessing the repository of model attributes on the blockchain network further comprises:
retrieving the first artificial intelligence model network identifier from the first request; searching, using a blockchain network explorer, the blockchain network for the firs

Assignees

• Capital One Services Llc

Inventors

• Sharpe Samuel
• Rafferty Galen
• Barr Brian
• Walters Austin
• Goodsitt Jeremy
• Bean Kenny

Classifications

• H04L41/16

  using machine learning or artificial intelligence · CPC title

• H04L67/306 Primary

  User profiles · CPC title

• H04L9/50 Primary

  using hash chains, e.g. blockchains or hash trees · CPC title

• H04L9/3239

  involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD · CPC title