Evaluation of modeling algorithms with continuous outputs

Patent metadata

Field	Value
Publication number	US-12475409-B2
Application number	US-202218052762-A
Country	US
Kind code	B2
Filing date	Nov 4, 2022
Priority date	Oct 31, 2018
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

Certain aspects involve evaluating modeling algorithms whose outputs can impact machine-implemented operating environments. For instance, a computing system generates, from a comparison of a set of estimated attribute values of an attribute to a set of validation attribute values of the attribute, a discretized evaluation dataset with data values in multiple categories. The computing system computes, for a modeling algorithm used to generate the estimated attribute values, an evaluation metric. The computing system provides a host computing system with access to the evaluation metric, one or more modeling outputs generated with the modeling algorithm, or both. Providing one or more of these outputs to the host computing system can facilitate modifying one or more machine-implemented operations.

First claim

What is claimed is:

1 . A system comprising:
a data repository storing data samples having values of variables for input to a machine-learning model for risk assessment for an entity, an external-facing subsystem configured for preventing a host server system from accessing the data repository via a data network, and
an evaluation system configured for:
accessing (a) an estimated dataset having a set of estimated values of an attribute that is a continuous variable, the estimated dataset generated by applying the machine-learning model to an input dataset of the data samples and (b) a validation dataset having a set of validation values of the attribute, the set of validation values respectively being known values corresponding to the set of estimated values generated by the machine-learning model,
generating, from a comparison of the estimated dataset and the validation dataset to an outcome of interest, a discretized evaluation dataset with data values in multiple categories, the discretized evaluation dataset comprising a set of categories in a classification matrix and a number of instances in each category, the set of categories including a true positive category, a true negative category, a false positive category, and a false negative category,
computing, for the machine-learning model, an evaluation metric based on a comparison of data values from different categories of the discretized evaluation dataset, the evaluation metric indicating an accuracy of the machine-learning model, and
providing the host server system with access to (a) the evaluation metric or (b) a modeling output generated with the machine-learning model which indicates a risk level associated with the entity, causing the host server system to allow or prevent the entity to access to a restricted function of a computing environment, based on the modeling output, wherein generating the discretized evaluation dataset comprises:
identifying a first category for the discretized evaluation dataset indicating a match between estimated attribute values and validation attribute values with respect to the outcome of interest;
identifying a second category for the discretized evaluation dataset indicating a mismatch between estimated attribute values and validation attribute values with respect to the outcome of interest;
determining, from the comparison of the estimated dataset and the validation dataset to the outcome of interest, a number of matches in the first category and a number of mismatches in the second category; and
outputting the discretized evaluation dataset having the first category with the number of matches and the second category with the number of mismatches.

2 . The system of claim 1 , wherein:
the outcome of interest comprises the attribute having a value greater than a threshold attribute value, the match comprises both a first estimated attribute value and a first validation attribute value being greater than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value, the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.

3 . The system of claim 1 , wherein:
the outcome of interest comprises the attribute having a value less than a threshold attribute value, the match comprises both a first estimated attribute value and a first validation attribute value being less than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value, the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second estimated attribute value and the second validation attribute value being less than the threshold attribute value, the second validation attribute value corresponding to the second estimated attribute value.

4 . The system of claim 1 , wherein:
the first category comprises the true positive category and the true negative category, and the second category comprises the false positive category and the false negative category.

5 . The system of claim 1 , wherein computing the evaluation metric comprises computing a percentage of matches within a sum of the matches in the first category and the mismatches in the second category.

6 . The system of claim 1 , wherein the allowing or preventing the access comprises modifying a functionality of an online interface provided to a user device associated with the entity.

7 . A method comprising:
generating, by a server system and from a comparison of an estimated dataset and a validation dataset to an outcome of interest, a discretized evaluation dataset with data values in multiple categories, the discretized evaluation dataset comprising a set of categories in a classification matrix and a number of instances in each category, the set of categories including a true positive category, a true negative category, a false positive category, and a false negative category, the estimated dataset having a set of estimated attribute values of an attribute and the validation dataset having a set of validation attribute values of the attribute, the set of validation attribute values respectively corresponding to the set of estimated attribute values; computing, by the server system, an evaluation metric based on a comparison of data values from different categories of the discretized evaluation dataset, the evaluation metric indicating an accuracy of a modeling algorithm; and providing a host computing system with access to (a) the evaluation metric or (b) a modeling output generated with the modeling algorithm, wherein providing the host computing system with access to the one or more of (a) the evaluation metric and (b) the modeling output causes the host computing system to modify a host system operation which indicates a risk level associated with an entity, allowing or preventing the entity to access to a restricted function of a computing environment, based the modeling output, wherein generating the discretized evaluation dataset comprises:
identifying a first category for the discretized evaluation dataset indicating a match between estimated attribute values and validation attribute values with respect to the outcome of interest;
identifying a second category for the discretized evaluation dataset indicating a mismatch between estimated attribute values and validation attribute values with respect to the outcome of interest;
determining, from the comparison of the estimated dataset and the validation dataset to the outcome of interest, a number of matches in the first category and a number of mismatches in the second category; and
outputting the discretized evaluation dataset having the first category with the number of matches and the second category with the number of mismatches.

8 . The method of claim 7 , wherein:
the outcome of interest comprises the attribute having a value greater than a threshold attribute value, the match comprises both a first estimated attribute value and a first validation attribute value being greater than the threshold attribute value, the first validation attribute value corresponding to the first estimated attribute value, the mismatch comprises one of a second estimated attribute value and a second validation attribute value being greater than the threshold attribute value and another of the second es

Assignees

• Equifax Inc

Inventors

• Liu Lefei
• Liu Peter
• Liu Jiawei
• Gao Peter
• Chang Vickey

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N20/00

  Machine learning · CPC title