Method and system for providing explanation of prediction generated by an artificial neural network model

Patent metadata

Field	Value
Publication number	US-11315008-B2
Application number	US-201916281904-A
Country	US
Kind code	B2
Filing date	Feb 21, 2019
Priority date	Dec 31, 2018
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

This disclosure relates to method and system for providing an explanation for a prediction generated by an artificial neural network (ANN) model for a given input data. The method may include receiving the given input data and the prediction generated by the ANN model. The ANN model may be built and trained for a target application. The method may further include determining a plurality of relevant portions of the given input data. For each of the plurality of relevant portions, the method may further include fetching a portional prediction and a portional prediction score generated by the ANN model, and determining a degree of influence score based on the portional prediction score and a comparison between the portional prediction and the prediction. The method may further include providing the explanation for the prediction based on the degree of influence score of each of the plurality of relevant portions.

First claim

What is claimed is:

1. A method of providing an explanation for a prediction generated by an artificial neural network (ANN) model for a given input data, the method comprising:
receiving, by a prediction explanation device, the given input data and the prediction generated by the ANN model, wherein the ANN model is built and trained for a target application;
determining, by the prediction explanation device, a plurality of relevant portions of the given input data;
for each of the plurality of relevant portions,
fetching, by the prediction explanation device, a portional prediction and a portional prediction score generated by the ANN model; and
determining, by the prediction explanation device, a degree of influence score based on the portional prediction score and a comparison between the portional prediction and the prediction; and

providing, by the prediction explanation device, the explanation for the prediction based on the degree of influence score of each of the plurality of relevant portions.

2. The method of claim 1 , wherein the given input data comprises at least one of text data, audio data, video data, and image data.

3. The method of claim 1 , wherein determining the plurality of relevant portions comprises:
segmenting the given input data into a plurality of portions; and
processing each of the plurality of portions to filter the plurality of relevant portions.

4. The method of claim 1 , wherein the target application comprises a text based application, wherein the ANN model comprises a recurrent neural network (RNN) model, and wherein each of the plurality of relevant portions comprises a relevant token from a tokenized text.

5. The method of claim 1 , wherein the providing the explanations further comprises determining a set of influential portions from among the plurality of relevant portions based on the degree of influence score of each of the plurality of relevant portions.

6. The method of claim 5 , further comprising retuning the ANN model based on the set of influential portions.

7. The method of claim 1 , wherein the providing the explanations comprises rendering each of the plurality of relevant portions along with the corresponding degree of influence score.

8. A system for providing an explanation for a prediction generated by an artificial neural network (ANN) model for a given input data, the system comprising:
a prediction explanation device comprising at least one processor and a computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving the given input data and the prediction generated by the ANN model, wherein the ANN model is built and trained for a target application;
determining a plurality of relevant portions of the given input data;
for each of the plurality of relevant portions,
fetching a portional prediction and a portional prediction score generated by the ANN model; and
determining a degree of influence score based on the portional prediction score and a comparison between the portional prediction and the prediction; and

providing the explanation for the prediction based on the degree of influence score of each of the plurality of relevant portions.

9. The system of claim 8 , wherein the given input data comprises at least one of text data, audio data, video data, and image data.

10. The system of claim 8 , wherein determining the plurality of relevant portions comprises:
segmenting the given input data into a plurality of portions; and
processing each of the plurality of portions to filter the plurality of relevant portions.

11. The system of claim 8 , wherein the target application comprises a text based application, wherein the ANN model comprises a recurrent neural network (RNN) model, and wherein each of the plurality of relevant portions comprises a relevant token from a tokenized text.

12. The system of claim 8 , wherein the providing the explanations further comprises determining a set of influential portions from among the plurality of relevant portions based on the degree of influence score of each of the plurality of relevant portions.

13. The system of claim 12 , wherein the operations further comprise retuning the ANN model based on the set of influential portions.

14. The system of claim 8 , wherein the providing the explanations comprises rendering each of the plurality of relevant portions along with the corresponding degree of influence score.

15. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a given input data and a prediction generated by an artificial neural network (ANN) model for the given input data, wherein the ANN model is built and trained for a target application;
determining a plurality of relevant portions of the given input data;
for each of the plurality of relevant portions,
fetching a portional prediction and a portional prediction score generated by the ANN model; and
determining a degree of influence score based on the portional prediction score and a comparison between the portional prediction and the prediction; and

providing an explanation for the prediction based on the degree of influence score of each of the plurality of relevant portions.

16. The non-transitory computer-readable medium of the claim 15 , wherein the target application comprises a text based application, wherein the ANN model comprises a recurrent neural network (RNN) model, and wherein each of the plurality of relevant portions comprises a relevant token from a tokenized text.

17. The non-transitory computer-readable medium of the claim 15 , wherein determining the plurality of relevant portions comprises:
segmenting the given input data into a plurality of portions; and
processing each of the plurality of portions to filter the plurality of relevant portions.

18. The non-transitory computer-readable medium of the claim 15 , wherein the providing the explanations further comprises determining a set of influential portions from among the plurality of relevant portions based on the degree of influence score of each of the plurality of relevant portions.

19. The non-transitory computer-readable medium of the claim 18 , wherein the operations further comprise retuning the ANN model based on the set of influential portions.

20. The non-transitory computer-readable medium of the claim 15 , wherein the providing the explanations comprises rendering each of the plurality of relevant portions along with the corresponding degree of influence score.

Assignees

• Wipro Ltd

Inventors

• Chatterjee Arindam
• Bandop Adhyay Tapati

Classifications

• G06N3/048 Primary

  Activation functions · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N5/045

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title