Faithfulness evaluations for model explanations often disagree due to differing assumptions
Navigating Disagreement Among Criteria for Evaluating Explanation Faithfulness
Machine LearningComputers and Society
Summary
The paper looks at how different ways to judge whether an AI explanation truly reflects the model can often disagree. The authors show these disagreements come from different choices about what exactly an explanation claims and how to measure agreement. They propose a framework that helps understand these differences and guides selecting the best approach depending on the real-world setting. This helps people who work with AI models make better and more justified decisions when checking if explanations are trustworthy.
What this means in practice
- •For model auditors: Assess explanation faithfulness by choosing evaluation methods fitting legal and organizational contexts using the paper’s criteria framework.
- •For ai compliance teams: Tailor explanation faithfulness evaluations for industry-required notifications by applying the paper’s analysis of operational choices and justification.
A position paper. It proposes an approach and reports no results.
Authors
Grant Shanklin, Jiří Němeček, Asia Biega
Abstract
Faithfulness requires that algorithmic explanations accurately reflect the model's behavior. Yet existing operationalizations of faithfulness can disagree about whether the same explanation is faithful, leaving developers and auditors to choose among conflicting evaluations. In this paper, we investigate disagreement among evaluation criteria and offer a practical path forward for evaluating explanation faithfulness. We first identify how operationalizations of faithfulness differ. We introduce a faithfulness functional that locates their disagreements in what an explanation claims about the model, which comparisons bear on that claim, and how agreement across those comparisons should be aggregated, and we reconstruct existing evaluation criteria within it. We argue these reconstructions support understanding faithfulness as a common though incompletely specified relation between the model and explanation, so the differences reflect choices left open by the general requirement. Justifying an operationalization then comes down to justifying these choices, which, on our view, draws on explanatory, institutional, and normative considerations. We support this argument with a case study of adverse action notices, reconstructing the evaluation implicit in an industry-proposed procedure and identifying which of its choices the setting justifies and which remain open. Our work gives auditors and practitioners a way to navigate disagreeing evaluations of faithfulness. It also explains why the literature should expect multiple operationalizations, while showing how concrete settings can narrow the choice among them. More broadly, our analysis shows how a shared requirement of faithfulness can support contextually justified evaluations despite disagreement among existing metrics.