Systems and methods for counterfactual explanation in machine learning models

Patent metadata

Field	Value
Publication number	US-2022114464-A1
Application number	US-202117162967-A
Country	US
Kind code	A1
Filing date	Jan 29, 2021
Priority date	Oct 8, 2020
Publication date	Apr 14, 2022
Grant date	—

Abstract

Embodiments described herein provide a two-stage model-agnostic approach for generating counterfactual explanation via counterfactual feature selection and counterfactual feature optimization. Given a query instance, counterfactual feature selection picks a subset of feature columns and values that can potentially change the prediction and then counterfactual feature optimization determines the best feature value for the selected feature as a counterfactual example.

First claim

What is claimed is:

1 . A method for generating a counterfactual example for counterfactual explanation in a machine learning model, the method comprising:
receiving a query instance including a plurality of feature columns and corresponding feature values; generating, by a machine learning model, a prediction result in response to the query instance; identifying, from the plurality feature columns, a subset of feature columns having a feature column associated with alternative feature values that would have potentially change the prediction result; determining an optimal policy that maximizes an objective comprising an expected feedback reward caused by a modified query instance; selecting a number of feature columns corresponding to highest feature values and a number of associated feature values according to the determined optimal policy; constructing a counterfactual example using the selected number of feature columns and the number of associated feature values; and outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance.

2 . The method of claim 1 , wherein the objective is determined by reinforcement learning (RL) in which:
feature columns and values to be modified are treated as an action in the RL; a counterfactual example candidate taking the action on the query instance is treated as a state in the RL; and a reward function of the RL is a prediction score of a different label other than the prediction result.

3 . The method of claim 2 , wherein the objective is subject to a sparsity constraint that is imposed on a stochastic policy for selecting the action.

4 . The method of claim 3 , wherein the stochastic policy is computed by a probability distribution parameterized by parameters of the machine learning model.

5 . The method of claim 1 , wherein the objective includes a first term based on the machine learning model output corresponding to a different label other than the prediction result and a variable indicating whether a particular feature column is to be modified.

6 . The method of claim 5 , wherein the objective is optimized by a gradientless descent procedure that encourages a sparse solution.

7 . The method of claim 1 , further comprising:
generating diverse counterfactual examples from the optimal policy.

8 . The method of claim 7 , wherein the diverse counterfactual examples are generated by:
constructing a set of counterfactual examples from the optimal policy; sorting the set of counterfactual examples based on respective objective function values; and removing duplicate counterfactual examples from the set of counterfactual examples.

9 . The method of claim 1 , wherein the machine learning model is a multivariate time series classifier that receives the query instance of a time series of input sequences.

10 . The method of claim 1 , wherein the subset of feature columns having a feature column associated with alternative feature values are identified from a number of nearest neighboring query instances to the query instance via a nearest neighbor search tree built from a training dataset.

11 . A system for generating a counterfactual example for counterfactual explanation in a machine learning model, the system comprising:
a memory that stores the machine learning model; a data interface that receives a query instance including a plurality of feature columns and corresponding feature values; and a processor that reads instructions from the memory to perform:
generating, by the machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns having a feature column associated with alternative feature values that would have potentially change the prediction result;
determining an optimal policy that maximizes an objective comprising an expected feedback reward caused by a modified query instance;
selecting a number of feature columns corresponding to highest feature values and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using the selected number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance.

12 . The system of claim 11 , wherein the objective is determined by reinforcement learning (RL) in which:
feature columns and values to be modified are treated as an action in the RL; a counterfactual example candidate taking the action on the query instance is treated as a state in the RL; and a reward function of the RL is a prediction score of a different label other than the prediction result.

13 . The system of claim 12 , wherein the objective is subject to a sparsity constraint that is imposed on a stochastic policy for selecting the action.

14 . The system of claim 13 , wherein the stochastic policy is computed by a probability distribution parameterized by parameters of the machine learning model.

15 . The system of claim 11 , wherein the objective includes a first term based on the machine learning model output corresponding to a different label other than the prediction result and a variable indicating whether a particular feature column is to be modified.

16 . The system of claim 15 , wherein the objective is optimized by a gradientless descent procedure that encourages a sparse solution.

17 . The system of claim 1 , wherein the processor further reads instructions from the memory to perform:
generating diverse counterfactual examples from the optimal policy.

18 . The system of claim 17 , wherein the diverse counterfactual examples are generated by:
constructing a set of counterfactual examples from the optimal policy; sorting the set of counterfactual examples based on respective objective function values; and removing duplicate counterfactual examples from the set of counterfactual examples.

19 . The system of claim 11 , wherein the machine learning model is a multivariate time series classifier that receives the query instance of a time series of input sequences.

20 . A processor-readable non-transitory storage medium storing processor-readable instructions for generating a counterfactual example for counterfactual explanation in a machine learning model, the instructions being executed by a processor to perform:
receiving a query instance including a plurality of feature columns and corresponding feature values; generating, by a machine learning model, a prediction result in response to the query instance; identifying, from the plurality feature columns, a subset of feature columns having a feature column associated with alternative feature values that would have potentially change the prediction result; determining an optimal policy that maximizes an objective comprising an expected feedback reward caused by a modified query instance; selecting a number of feature columns corresponding to highest feature values and a number of associated feature values according to the determined optimal policy; constructing a counterfactual example using the selected number of feature columns and the number of associated feature values; and outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result

Assignees

• Salesforce Com Inc

Inventors

• Yang Wenzhuo
• Li Jia
• Hoi Chu Hong
• Xiong Caiming

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/211

  Selection of the most significant subset of features · CPC title