Taking the Whys Seriously: Limitations of Counterfactual Explanations in Justification and Recourse
2026-08-31 • Computers and Society
Computers and SocietyArtificial Intelligence
AI summaryⓘ
The authors look at how counterfactual explanations (CEs), which show how changing input features would alter AI model outputs, are used in real-world situations. They find that simply using CEs to justify decisions or offer recourse can hide important choices made throughout the AI development process. Their experiments show that these earlier design decisions impact the explanations as much as the method creating them. The authors suggest that CEs alone don't fully answer why decisions were made, especially without considering if different choices should have been made by the decision-maker.
counterfactual explanationsexplainable AImachine learning pipelinemodel justificationalgorithmic recoursemodel validationfeature measurementdecision-makingbusiness requirements
Authors
Mattia Cerrato, Otto Sahlgren, Xenia Heilmann
Abstract
Counterfactual explanations (CEs) are widely used in explainable artificial intelligence (AI) to show how a model's outputs would change if the input features were manipulated. This technique is used for a range of tasks such as debugging models, explaining predictions, justifying decisions, and providing algorithmic recourse. In this paper, we explore the normative legitimacy of employing counterfactuals in real-life model deployment settings. We discuss the different stakes involved in these different purposes for which CEs are commonly employed, and find stricter requirements for justification and recourse. In particular, we find that naive application of CEs for justification and recourse can lead to ignoring contestable choices made throughout the machine learning (ML) pipeline, thus obfuscating that decisions and counterfactuals for those decisions are also artifacts of an organization's materialized design and governance choices. We demonstrate this with four empirical experiments involving interventions at stages of the ML pipeline ``upstream" of the explanation itself, and show that these affect the generated counterfactuals. We find that an organization's choices on measurement models for feature and labels, business requirements, model validation, and the metric of model success have as much or more impact on the generated counterfactuals as the specifics of the generating method. Our findings underline the need to account for such choices upon providing justification and recourse, providing a stark reminder of the relational nature of these tasks. As putative justifications or recourse recommendations, CEs do not provide adequate answers to some important "why"-questions because they preclude consideration of whether the decision-maker ought to have acted differently.