Monitoring machine learning models using surrogate model output

Patent metadata

Field	Value
Publication number	US-2024119303-A1
Application number	US-202218045746-A
Country	US
Kind code	A1
Filing date	Oct 11, 2022
Priority date	Oct 11, 2022
Publication date	Apr 11, 2024
Grant date	—

Abstract

In some aspects, a computing system may use a surrogate machine learning model to detect whether a production or other machine learning model has a tendency to generate different output depending on which subpopulation a particular sample belongs to. The surrogate machine learning model may be trained using features/outputs that are not included in the data used by the production model. For example, by using demographic information in lieu of the original labels of a dataset that was used to train a production model, a surrogate model may be used to detect whether the production model is able to discern one or more characteristics associated with but not present in a sample using other features of the dataset. Output of the surrogate machine learning model may be clustered to detect whether certain subpopulations are treated differently by the production model.

First claim

What is claimed is:

1 . A system for using a surrogate machine learning model to detect and remove bias in output generated by a production machine learning model, the system comprising:
one or more processors; and a non-transitory, computer-readable medium having instructions recorded thereon that, when executed by the one or more processors, cause operations comprising:
generating a surrogate dataset for training a surrogate machine learning model such that: (i) the surrogate dataset is a modified version of a production training dataset used to train a production machine learning model to classify an input with a respective label of a production set of labels included in the production training dataset; and (ii) the surrogate dataset comprises a surrogate set of labels in lieu of the production set of labels of the production training dataset, wherein the surrogate set of labels is not included in the production training dataset;
using the surrogate dataset to train the surrogate machine learning model to classify an input with a respective label of the surrogate set of labels;
obtaining production inputs and production outputs of the production machine learning model, each production output of the production outputs comprising a respective label of the production set of labels that was generated via the production machine learning model in response to providing the production machine learning model with a corresponding production input of the production inputs;
providing, to the surrogate machine learning model trained on the surrogate dataset, the production inputs to obtain surrogate outputs corresponding to the production outputs, each surrogate output of the surrogate outputs comprising (i) a respective surrogate label of the surrogate set of labels and (ii) a confidence score for the respective surrogate label;
performing clustering based on confidence scores of the surrogate outputs to determine one or more clusters indicating a threshold-exceeding correlation between (i) a respective label of the surrogate set of labels, (ii) a respective label of the production set of labels, and (iii) one or more respective feature values of features of the production inputs; and
generating, based on the one or more clusters, an indication of a modification related to the production machine learning model.

2 . A method comprising:
obtaining (i) first inputs provided to a first machine learning model and (ii) first outputs generated via the first machine learning model based on the first inputs, the first machine learning model being trained to classify an input with a respective label of a first set of labels; accessing a second machine learning model trained on a surrogate dataset comprising a second set of labels in lieu of the first set of labels, the second machine learning model being trained to classify an input with a respective label of the second set of labels; providing, to the second machine learning model, the first inputs to obtain second outputs corresponding to the first outputs, each second output of the second outputs comprising a respective classification corresponding to the second set of labels; determining one or more clusters indicating a correlation between (i) a respective label of the second set of labels, (ii) a respective label of the first set of labels, and (iii) one or more respective feature values of features of the first inputs; and generating, based on the one or more clusters, an indication of a modification related to the first machine learning model.

3 . The method of claim 2 , wherein determining one or more clusters comprises:
separating each classification of the second outputs based on a threshold confidence score, wherein confidence scores of a first cluster of the one or more clusters are below the threshold confidence score and confidence scores of a second cluster of the one or more clusters are above the threshold confidence score.

4 . The method of claim 3 , further comprising:
determining that the first machine learning model outputs more than a threshold number of classifications of a first type for the first cluster and outputs fewer than the threshold number of classifications of the first type for the second cluster; and based on the first machine learning model outputting more than the threshold number of classifications of the first type for the first cluster and outputting fewer than the threshold number of classifications of the first type for the second cluster, determining that the first cluster is anomalous.

5 . The method of claim 2 , wherein determining the one or more clusters comprises:
inputting the output of the second machine learning model into a clustering model; and determining, via the clustering model, the one or more clusters.

6 . The method of claim 2 , wherein the correlation indicates bias in the first machine learning model, the bias being associated with the second set of labels, wherein the second set of labels is not in the first inputs.

7 . The method of claim 6 , wherein generating the indication of a modification related to the first machine learning model comprises:
modifying output of the first machine learning model such that the bias is no longer present in the output.

8 . The method of claim 2 , further comprising:
generating a user interface for displaying an indication of the modification and output generated by the first machine learning model; and causing display of the user interface.

9 . The method of claim 2 , wherein a confidence score associated with the second outputs indicates a level of certainty that a corresponding classification is correct.

10 . The method of claim 2 , wherein the first set of labels indicates whether an anomaly has been detected, the second set of labels indicates locations of computing devices, and wherein the second machine learning model is trained to determine a location of a computing device.

11 . The method of claim 2 , wherein the surrogate dataset further comprises output generated via the first model, and wherein the output generated by the second machine learning model comprises a counterfactual sample corresponding to the output generated via the first model.

12 . A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause operations comprising:
obtaining (i) first inputs provided to a first machine learning model and (ii) first outputs generated via the first machine learning model based on the first inputs, the first machine learning model being trained to classify an input with a respective label of a first set of labels; accessing a second machine learning model trained on a surrogate dataset comprising a second set of labels in lieu of the first set of labels, the second machine learning model being trained to classify an input with a respective label of the second set of labels; providing, to the second machine learning model, the first inputs to obtain second outputs corresponding to the first outputs, each second output of the second outputs comprising a respective classification corresponding to the second set of labels; determining one or more clusters indicating a correlation between (i) a respective label of the second set of labels, (ii) a respective label of the first set of labels, and (iii) one or more respective feature values of features of the first inputs; and generating, based on the one or more clusters, an indication of a modification related to the first machine learning model.

13 . The medium of claim 12 , wherein determining one or more clusters comprises:
separating each classification of the

Assignees

• Capital One Services Llc

Inventors

• Sharpe Samuel
• Bruss Christopher Bayan
• Barr Brian
• Au-Yeung Justin

Classifications

• G06N3/096 Primary

  Transfer learning · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title