Diagnosing anomalies detected by black-box machine learning models

Patent metadata

Field	Value
Publication number	US-11487650-B2
Application number	US-202016882430-A
Country	US
Kind code	B2
Filing date	May 22, 2020
Priority date	May 22, 2020
Publication date	Nov 1, 2022
Grant date	Nov 1, 2022

Abstract

A computer-implemented method, a computer program product, and a computer system for diagnosing anomalies detected by a black-box machine learning model. A computer determines a local variance of a test sample in a test dataset, where the local variance represents uncertainty of a prediction by the black-box machine learning model. The computer initializes optimal compensations for the test sample, where the optimal compensations are optimal perturbations to test sample values of respective components of a multivariate input variable. The computer determines local gradients for the test sample. Based on the local variance and the local gradients, the computer updates the optimal compensations until convergences of the optimal compensations are reached. Using the optimal compensations, the computer diagnoses the anomalies detected by the black-box machine learning model.

First claim

What is claimed is:

1. A computer-implemented method for diagnosing anomalies which are deviations from predictions by a black-box machine learning model, the method comprising:
receiving, by a computer, a test dataset for the black-box machine learning model, wherein each test sample in the test dataset includes test sample values of respective components of a multivariate input variable and a test sample value of an output variable;
determining, by the computer, a local variance of each test sample in the test dataset, the local variance representing uncertainty of a prediction by the black-box machine learning model, wherein the local variance of each test sample is computed based on a difference between predicted values by the black-box machine learning model and test sample values of the output variable;
initializing, by the computer, for each test sample in the test dataset, optimal compensations, wherein the optimal compensations are defined as optimal perturbations needed to achieve a highest likelihood in a vicinity of the test sample values of the respective components of the multivariate input variable and the optimal compensations represent deviations between multivariate input variable values by which the black-box machine learning model predicts a value being equal to the test sample value of the output variable and the test sample values of the respective components of the multivariate input variable;
determining, by the computer, for each test sample in the test dataset, a local gradient of the black-box machine learning model at a point with values that are summations of the test sample values of the respective components of the multivariate input variable and values of the optimal compensations, wherein the local gradient is with respect to the optimal compensations;
updating, by the computer, for each test sample in the test dataset, the optimal compensations, based on the local variance and the local gradient;
determining, by the computer, for each test sample in the test dataset, whether the optimal compensations converge;
in response to determining that the optimal compensations converge, obtaining, by the computer, for each test sample in the test dataset, final values of the optimal compensations; and
diagnosing, by the computer, which one or more components in the multivariate input variable are responsible for the anomalies, using the final values of the optimal compensations.

2. The computer-implemented method of claim 1 , further comprising:
in response to determining that the optimal compensations do not converge, reiterating, by the computer, determining the local gradient, updating the optimal compensations, and determining whether the optimal compensations converge, until convergences of the optimal compensations are reached.

3. The computer-implemented method of claim 1 , wherein the optimal compensations are initialized with negligible random numbers which are close to zero.

4. The computer-implemented method of claim 1 , wherein the local gradient is computed by local linear fitting and the local gradient is obtained as regression coefficients.

5. The computer-implemented method of claim 1 , further comprising:
in response to determining that a deviation between the test sample value of the output variable and a predicted value by the black-box machine learning model is less than a predetermined threshold, determining, by the computer, that convergences of the optimal compensations are reached; and
wherein the predicted value is calculated at the point with the values that are the summations of the test sample values of the respective components of the multivariate input variable and values of the optimal compensations.

6. The computer-implemented method of claim 1 , further comprising:
in response to determining that the local gradient is less than a predetermined threshold, determining, by the computer, that convergences of the optimal compensations are reached.

7. The computer-implemented method of claim 1 , further comprising:
receiving, by the computer, parameters for determining the optimal compensations; and
wherein the parameters includes a hyperparameter controlling an overall scale of the optimal compensations, a hyperparameters controlling a sparsity of the optimal compensations, and a hyperparameter representing a learning rate.

8. A computer program product for diagnosing anomalies which are deviations from predictions by a black-box machine learning model, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors, the program instructions executable to:
receive, by a computer, a test dataset for the black-box machine learning model, wherein each test sample in the test dataset includes test sample values of respective components of a multivariate input variable and a test sample value of an output variable;
determine, by the computer, a local variance of each test sample in the test dataset, the local variance representing uncertainty of a prediction by the black-box machine learning model, wherein the local variance of each test sample is computed based on a difference between predicted values by the black-box machine learning model and test sample values of the output variable;
initialize, by the computer, for each test sample in the test dataset, optimal compensations, wherein the optimal compensations are defined as optimal perturbations needed to achieve a highest likelihood in a vicinity of the test sample values of the respective components of the multivariate input variable and the optimal compensations represent deviations between multivariate input variable values by which the black-box machine learning model predicts a value being equal to the test sample value of the output variable and the test sample values of the respective components of the multivariate input variable;
determine, by the computer, for each test sample in the test dataset, a local gradient of the black-box machine learning model at a point with values that are summations of the test sample values of the respective components of the multivariate input variable and values of the optimal compensations, wherein the local gradient is with respect to the optimal compensations;
update, by the computer, for each test sample in the test dataset, the optimal compensations, based on the local variance and the local gradient;
determine, by the computer, for each test sample in the test dataset, whether the optimal compensations converge;
in response to determining that the optimal compensations converge, obtain, by the computer, for each test sample in the test dataset, final values of the optimal compensations; and
diagnose, by the computer, which one or more components in the multivariate input variable are responsible for the anomalies, using the final values of the optimal compensations.

9. The computer program product of claim 8 , further comprising the program instructions executable to:
in response to determining that the optimal compensations do not converge, reiterate, by the computer, determining the local gradient, updating the optimal compensations, and determining whether the optimal compensations converge, until convergences of the optimal compensations are reached.

10. The computer program product of claim 8 , wherein the optimal compensations are initialized with negligible random numbers which are close to zero.

11. The computer program product of claim 8 , wherein the local gradient is computed by local linear fitting and the local gradient is obtained as regression coefficients.

12. The computer pr

Assignees

• IBM

Inventors

• Ide Tsuyoshi
• Dhurandhar Amit
• Navratil Jiri
• Abe Naoki
• Singh Moninder

Classifications

• G06N5/045

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06F11/3457

  Performance evaluation by simulation · CPC title