Systems and methods for counterfactual explanation in machine learning models

Patent metadata

Field	Value
Publication number	US-12400136-B2
Application number	US-202117162967-A
Country	US
Kind code	B2
Filing date	Jan 29, 2021
Priority date	Oct 8, 2020
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

Embodiments described herein provide a two-stage model-agnostic approach for generating counterfactual explanation via counterfactual feature selection and counterfactual feature optimization. Given a query instance, counterfactual feature selection picks a subset of feature columns and values that can potentially change the prediction and then counterfactual feature optimization determines the best feature value for the selected feature as a counterfactual example.

First claim

What is claimed is:

1. A method for generating a counterfactual example for counterfactual explanation in a machine learning model, the method comprising:
receiving a query instance including a plurality of feature columns and corresponding feature values;
generating, by a machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns associated with corresponding alternative feature values that would have potentially changed the prediction result;
determining an optimal policy that maximizes an expected feedback reward earned for modifying the query instance including changing a feature value of at least one feature column of the subset of feature columns;
determining a number of feature columns of the subset of feature columns and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using a portion of the determined number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance, wherein the optimal policy is determined by reinforcement learning (RL) in which:
changing the feature value of the at least one feature column is treated as an action in the RL;
the modified query instance is treated as a state in the RL; and
a reward function of the RL for determining the expected feedback reward is a prediction score of a different label other than the prediction result.

2. The method of claim 1 , wherein the optimal policy is determined subject to a sparsity constraint for selecting the action.

3. The method of claim 2 , wherein the sparsity constraint is imposed via a probability distribution parameterized by parameters of the machine learning model.

4. The method of claim 1 , wherein the optimal policy is determined by a gradientless descent procedure that encourages a sparse solution for selecting the action and a gradient descent procedure that encourages an exploration solution for selecting the action.

5. The method of claim 1 , wherein:
the counterfactual example is a first counterfactual example; and
the constructing comprises constructing a diverse set of counterfactual examples including the first counterfactual example and a second counterfactual example using at least one feature column different from the portion of the determined number of feature columns.

6. The method of claim 5 , further comprising:
sorting the set of counterfactual examples based on respective objective function values; and
removing duplicate counterfactual examples from the set of counterfactual examples.

7. The method of claim 1 , wherein the machine learning model is a multivariate time series classifier that receives the query instance of a time series of input sequences.

8. The method of claim 1 , wherein the subset of feature columns are identified from a number of nearest neighboring query instances to the query instance via a nearest neighbor search tree built from a training dataset.

9. A system for generating a counterfactual example for counterfactual explanation in a machine learning model, the system comprising:
a memory that stores the machine learning model;
a data interface that receives a query instance including a plurality of feature columns and corresponding feature values; and
a processor that reads instructions from the memory to perform:
generating, by the machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns associated with corresponding alternative feature values that would have potentially changed the prediction result;
determining an optimal policy that maximizes an expected feedback reward earned for modifying the query instance including changing a feature value of at least one feature column of the subset of feature columns;
determining a number of feature columns of the subset of feature columns and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using a portion of the determined number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counterfactual explanation for the prediction result in response to the query instance, wherein the optimal policy is determined by reinforcement learning (RL) in which:
changing the feature value of the at least one feature column is treated as an action in the RL;
the modified query instance is treated as a state in the RL; and
a reward function of the RL for determining the expected feedback reward is a prediction score of a different label other than the prediction result.

10. The system of claim 9 , wherein the optimal policy is determined subject to a sparsity constraint that is imposed on a stochastic policy for selecting the action.

11. The system of claim 10 , wherein the sparsity constraint is imposed via a probability distribution parameterized by parameters of the machine learning model.

12. The system of claim 9 , wherein the optimal policy is determined by a gradientless descent procedure that encourages a sparse solution for selecting the action and a gradient descent procedure that encourages an exploration solution for selecting the action.

13. The system of claim 9 , wherein the counterfactual example is a first counterfactual example and the processor further reads instructions from the memory to:
construct a diverse set of counterfactual examples including the first counterfactual example and a second counterfactual example using at least one feature column different from the portion of the determined number of feature columns.

14. The system of claim 13 , wherein the processor further reads instructions from the memory to perform:
sorting the set of counterfactual examples based on respective objective function values; and
removing duplicate counterfactual examples from the set of counterfactual examples.

15. The system of claim 9 , wherein the machine learning model is a multivariate time series classifier that receives the query instance of a time series of input sequences.

16. A processor-readable non-transitory storage medium storing processor-readable instructions for generating a counterfactual example for counterfactual explanation in a machine learning model, the instructions being executed by a processor to perform:
receiving a query instance including a plurality of feature columns and corresponding feature values;
generating, by a machine learning model, a prediction result in response to the query instance;
identifying, from the plurality feature columns, a subset of feature columns associated with corresponding alternative feature values that would have potentially changed the prediction result;
determining an optimal policy that maximizes an expected feedback reward earned for modifying the query instance including changing a feature value of at least one feature column of the subset of feature columns;
determining a number of feature columns of the subset of feature columns and a number of associated feature values according to the determined optimal policy;
constructing a counterfactual example using a portion of the determined number of feature columns and the number of associated feature values; and
outputting, by the machine learning model, the counterfactual example as a counte

Assignees

• Salesforce Inc

Inventors

• Yang Wenzhuo
• Li Jia
• Hoi Chu Hong
• Xiong Caiming

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06F18/24323

  Tree-organised classifiers · CPC title