Explainers for machine learning classifiers

Patent metadata

Field	Value
Publication number	US-10824959-B1
Application number	US-201615045030-A
Country	US
Kind code	B1
Filing date	Feb 16, 2016
Priority date	Feb 16, 2016
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

A transformed data set corresponding to a machine learning classifier's training data set is generated. Each transformed record contains a modified version of a corresponding training record, as well as the prediction made for the training record by the classifier. A set of explanatory rules is minded from the transformed data set, with each rule indicating a relationship between the prediction and one or more features corresponding to the training records. From among the rule set, a particular matching rule is selected to provide an easy-to-understand explanation for a prediction made by the classifier for an observation record which is not part of the training set.

First claim

What is claimed is:

1. A system, comprising:
one or more computing devices of a machine learning service implemented at a provider network;
wherein the one or more computing devices are configured to:
receive a first request, via a programmatic interface from a client, to generate a classification model with respect to an input data set comprising a plurality of observation records, wherein individual ones of the observation records comprise respective values of (a) a plurality of input variables and (b) an output variable;
train, using a selected classification algorithm, the classification model based at least in part on an analysis of a training set of observation records of the input data set;
generate, corresponding to at least a subset of observation records of the training set, a transformed data set comprising a plurality of intermediate data records, wherein a particular intermediate data record of the transformed data set comprises: (a) a binarized feature derived from a corresponding non-binary attribute value of a particular observation record of the training set and (b) a prediction produced by the classification model for the output variable with respect to the particular observation record;
identify, using a selected rule mining algorithm on the transformed data set, a plurality of explanatory rules, wherein a first explanatory rule of the plurality of explanatory rules indicates an implied relationship between a prediction result of the classification model and a predicate on a particular attribute of the observation records;
rank the plurality of explanatory rules based on one or more ranking metrics; and
in response to a request for an explanation of a particular prediction produced by the classification model with respect to a second observation record, provide a representation of a particular explanatory rule, wherein the particular explanatory rule is selected from the plurality of explanatory rules based at least in part on (a) a rank of the particular explanatory rule and (b) a result of a comparison between a predicate of the particular explanatory rule and the second observation record.

2. The system as recited in claim 1 , wherein the first request indicates the selected classification algorithm.

3. The system as recited in claim 1 , wherein the selected classification algorithm comprises one or more of: (a) a neural network algorithm, (b) a random forest algorithm, or (c) a boosted gradient tree algorithm.

4. The system as recited in claim 1 , wherein the selected rule mining algorithm comprises one or more of: (a) an Apriori algorithm, (b) an FP-Growth (Frequent Pattern-Growth) algorithm, (c) a CPAR (Classification based on Predictive Association Rules) algorithm, (d) a CN2 induction algorithm, (e) a CMAR (Classification based on Multiple Association Rules) algorithm, (f) a SLIPPER (Simple Learner with Iterative Pruning to Produce Error Reduction) algorithm, (g) a CART (Classification and Regression Tree) algorithm, (h) an ID3 (Iterative Dichotomizer 3) algorithm, (i) a C4.5 algorithm or (j) a C5.0 algorithm.

5. The system as recited in claim 1 , wherein the one or more computing devices are configured to:
in response to the first request, select, based at least in part on an examination of a knowledge base entry of the machine learning service, one or more of (a) a classification algorithm or (b) a rule-mining algorithm.

6. A method, comprising:
performing, at one or more computing devices:
identifying a training set of observation records used to generate a classification model;
determining, corresponding to at least a subset of observation records of the training set, a transformed data set comprising a plurality of intermediate data records, wherein a particular intermediate data record of the transformed data set comprises: (a) a feature derived from a corresponding attribute value of a particular observation record of the training set and (b) a prediction produced by the classification model for an output variable with respect to the particular observation record;
generating, using a selected rule mining algorithm on the transformed data set, an explainer corresponding to the classification model, wherein a first explanatory rule of a plurality of explanatory rules of the explainer indicates a relationship between a prediction result of the classification model and a predicate on a particular feature corresponding to one or more observation records;
in response to a request for an explanation of a particular prediction produced by the classification model with respect to a second observation record, providing a representation of a particular explanatory rule selected from the plurality of explanatory rules based at least in part on a result of a rank of the particular explanatory rule and a match operation with respect to a predicate of the particular explanatory rule and the second observation record.

7. The method as recited in claim 6 , wherein during training of the classification model, an internal representation corresponding to at least a portion of the training set is generated, wherein the particular intermediate data record comprises the internal representation.

8. The method as recited in claim 7 , wherein the classification model comprises a neural network model, and wherein the internal representation comprises a particular node of a neural network.

9. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
determining, corresponding to at least a subset of observation records of a second training set used to train a second classification model, a second transformed data set comprising a second plurality of intermediate data records, wherein a first intermediate data record of the second transformed data set comprises: (a) a feature derived from a corresponding attribute value of a first observation record of the second training set, (b) an internal representation, produced by the second classification model, of at least a portion of the first observation record and (c) a prediction produced by the classification model for an output variable of the first observation record;
generating, using the second transformed data set, a second explainer corresponding to the second classification model, wherein a first explanatory rule of a plurality of explanatory rules of the second explainer indicates a relationship between (a) a prediction produced by the second classification model and (b) a property of internal representations, produced by the second classification model, of respective observation records of the second training set; and
in response to a request for an explanation of a particular prediction produced by the second classification model, providing a first-level explanation indicating a particular explanatory rule selected from the plurality of explanatory rules of the second explainer.

10. The method as recited in claim 9 , further comprising performing, by the one or more computing devices:
in response to a request for an additional explanation, to be expressed in terms of input variables of observation records, of the particular prediction produced by the second classification model, provide an indication of at least another explanatory rule relating the property of the internal representations and a particular input variable.

11. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
ranking the plurality of explanatory rules based on one or more ranking metrics; and
selecting said particular explanatory rule from the plurality of explanatory rules based at least in part on a result of said ranking.

12. The meth

Assignees

• Amazon Tech Inc

Inventors

• Chatterjee Bibaswan Kumar
• Rao Srinivasan Sengamedu Hanumantha

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N5/045

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title

• G06N5/025 Primary

  Extracting rules from data · CPC title