Robust counterfactual explanations respond differently to varied model changes
Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations
Machine LearningArtificial Intelligence
Summary
Counterfactual explanations help people understand and change decisions made by AI models, but those explanations might stop working if the AI model changes. The authors show that different types of model updates, like small tweaks or complete retraining, affect explanations in different ways. They tested six methods designed to keep explanations robust across eight types of model changes and found that no single method works best for all. They suggest evaluating these methods using a standard approach that compares model changes and explanation reliability fairly.
What this means in practice
- •For machine learning engineers: Assess how stable counterfactual explanations remain across different model updates to maintain trustworthy AI decision support.
- •For data science teams: Choose and evaluate methods for generating explanations that remain valid when predictive models are retrained or modified.
Authors
Marcin Kostrzewa, Maciej Zięba
Abstract
Robust counterfactual explanations promise recourse that still works after the model behind it changes. Whether they keep that promise depends on what the change is. A small perturbation of the parameters, retraining on new data, and a new architecture are different events, and each existing method is evaluated against the one it was built for. Reported robustness scores, therefore, answer different questions and cannot be compared. We propose a unified cross-family evaluation protocol that holds factual instances and generated counterfactuals fixed while testing every method against the same eight types of model change. The benchmark compares six robust methods and two standard baselines on four tabular datasets. It characterizes every changed classifier through its outputs and reports empirical robustness together with coverage, base validity, and proximity. We find that relative performance and failure modes vary across change families. Bounded parameter perturbations change 0.95\% of test predictions on average, compared with 4.9\% for bootstrap retraining. Methods with guarantees for these perturbations do not necessarily transfer to other changes. RobX transfers most consistently in our experiments, although greater stability can require larger interventions. We argue that robust CFE methods should be evaluated through a common protocol that specifies the model changes, measures their realized behavioral magnitude, and keeps generation performance separate from robustness.