Method and system for global explainability of neural networks

Patent metadata

Field	Value
Publication number	US-2023196062-A1
Application number	US-202117555234-A
Country	US
Kind code	A1
Filing date	Dec 17, 2021
Priority date	Dec 17, 2021
Publication date	Jun 22, 2023
Grant date	—

Abstract

The layers of a neural network model are traversed in sequence one or more times while generating a plurality of relevance scores each time based on neuron weights and neuron biases of the neuron network model. Each relevance score of the plurality of relevance scores quantifies a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers. One or more relevance vectors can be populated from the plurality of relevance scores generated at the one or more times. Each of the relevance scores in each relevance vector quantifies a relevance of one of the input features to a task for which the neural network model is trained to perform. An explanation of a behavior of the neural network as a whole is generated based on the one or more relevance vectors.

First claim

1 . A computer-implemented method comprising:
receiving a request identifying a neural network model, the neural network model comprising a plurality of neurons arranged in a sequence of layers, a plurality of neuron weights, a plurality of neuron biases, and an input layer configured to receive an input vector with a plurality of input features; in response to receiving the request, traversing the sequence of layers one or more times while generating a plurality of relevance scores each time based on the neuron weights and neuron biases, each relevance score of the plurality of relevance scores quantifying a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers; generating a global explainability dataset comprising one or more relevance vectors populated with relevance scores from the plurality of relevance scores generated at the one or more times, each of the relevance scores in each relevance vector quantifying a relevance of one of the input features to a task the neural network model is trained to perform; and generating a global explanation of the neural network model based on the global explainability dataset.

2 . The method of claim 1 , wherein the sequence of layers comprises a plurality of hidden layers and a last layer, wherein a last hidden layer of the plurality of hidden layers precedes the last layer, wherein a first hidden layer of the plurality of hidden layers succeeds the input layer, and wherein the sequence of layers is traversed in a reverse direction from the last layer to the first hidden layer.

3 . The method of claim 2 , wherein generating the plurality of relevance scores comprises computing one or more relevance scores at the last layer based on neuron weights in the last layer and neuron biases in the last hidden layer.

4 . The method of claim 3 , wherein each relevance score computed at the last layer is a linear combination of a weight term and a bias term, wherein the weight term comprises a neuron weight in the last layer that connects a select neuron in the last hidden layer to a select neuron in the last layer, and wherein the bias term comprises a neuron bias connected to the select neuron in the last hidden layer.

5 . The method of claim 3 , wherein generating the plurality of relevance scores further comprises computing one or more relevance scores at the plurality of hidden layers, wherein the one or more relevance scores are computed at each one of the hidden layers based on neuron weights in the each one of the hidden layers, relevance scores in the higher layer succeeding the each one of the hidden layers, and neuron biases in a lower layer preceding the each one of the hidden layers.

6 . The method of claim 5 , wherein each relevance score computed at the each one of the hidden layers is a linear combination of a weighted relevance term and a bias term, wherein the weighted relevance term is based on the neuron weights in the each one of the hidden layers and the relevance scores in the higher layer succeeding the each one of the hidden layers, and wherein the bias term is based on the neuron biases in the lower layer preceding the each one of the hidden layers.

7 . The method of claim 5 , further comprising discarding the relevance scores computed at a higher layer succeeding the each one of the hidden layers after computing the relevance scores at the each one of the hidden layers.

8 . The method of claim 5 , wherein each of the one or more relevance vectors is populated with the one or more relevance scores computed at the first hidden layer during the corresponding time of traversing the sequence of layers.

9 . The method of claim 2 , wherein the last layer comprises a single neuron, wherein the sequence of layers is traversed one time corresponding to the single neuron, and wherein one relevance vector is populated using a subset of the plurality of relevance scores generated during the one time of traversing the sequence of layers.

10 . The method of claim 2 , wherein the last layer comprises a plurality of neurons, wherein the sequence of layers is traversed a plurality of times corresponding to the plurality of neurons, and wherein the global explainability dataset comprises a plurality of relevance vectors corresponding to the plurality of neurons in the last layer.

11 . The method of claim 10 , wherein each of the plurality of relevance vectors is populated using a subset of the plurality of relevance scores generated during the corresponding time of traversing the sequence of layers.

12 . The method of claim 1 , wherein the neural network model is a trained neural network model, and further comprising retraining the neural network model based at least in part on the global explanation.

13 . The method of claim 1 , wherein the neural network model is a trained neural network model, and further comprising receiving a modification to the global explanation and retraining the neural network model based at least in part on the modification to the global explanation.

14 . The method of claim 1 , further comprising storing the global explainability dataset in a data storage in association with the neural network model.

15 . The method of claim 1 , wherein the plurality of relevance scores are generated each time without using neuron activations of the neurons in the sequence of layers.

16 . One or more non-transitory computer-readable storage media storing computer-executable instructions for causing a computer system to perform operations comprising:
receiving a request identifying a neural network model, the neural network model comprising a plurality of neurons arranged in a sequence of layers, a plurality of neuron weights, a plurality of neuron biases, and an input layer configured to receive an input vector with a plurality of input features; in response to receiving the request, traversing the sequence of layers one or more times while generating a plurality of relevance scores each time based on the neuron weights and neuron biases, each relevance score of the plurality of relevance scores quantifying a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers; populating one or more relevance vectors from the plurality of relevance scores generated at the one or more times, each of the relevance scores in each relevance vector quantifying a relevance of one of the input features to a task the neural network model is trained to perform; and generating a global explanation of the neural network model based on the one or more relevance vectors.

17 . The one or more non-transitory computer-readable storage media of claim 16 , wherein the sequence of layers comprises a plurality of hidden layers and a last layer, wherein a last hidden layer of the plurality of hidden layers precedes the last layer, wherein a first hidden layer of the plurality of hidden layers succeeds the input layer, and wherein the sequence of layers is traversed in a reverse direction from the last layer to the first hidden layer.

18 . The one or more non-transitory computer-readable storage media of claim 17 , wherein generating the plurality of relevance scores comprises:
computing one or more relevance scores at the last layer based on neuron weights in the last layer and neuron biases in the last hidden layer; computing one or more relevance scores at each one of the hidden layers based on neuron weights in the each one of the hidden layers, relevance scores in the higher layer succeeding the each one of the hidden

Assignees

• Sap Se

Inventors

• Farooqi Waqas Ahmad
• Schmidt Eckehard
• Grill Jonas Benedict

Classifications

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title

• G06N5/045 Primary

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title