Limitations found in automated evaluation methods for AI explanations
Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations
Computation and LanguageMachine Learning
Summary
Evaluating how well AI explanations help people understand model decisions is important but time-consuming. The authors study automated methods that use large language models (LLMs) to simulate humans evaluating explanations. They find that these simulators often rely on shortcuts, like guessing answers from class names, rather than truly using the explanations. This means automated evaluation scores can be misleading. The authors suggest ways to improve these automated tests to make them more reliable.
What this means in practice
- •For machine learning developers: Improve the design of automated explanation evaluation tools by avoiding shortcuts that distort results.
- •For ai fairness auditors: Better assess explanation methods in classification systems to ensure explanations are genuinely informative.
Authors
Antonin Poché, Fanny Jourdan, Nils Feldhus, Qianli Wang, Jing Yang, Simon Ostermann, Nicholas Asher, Philippe Muller, Vera Schmitt
Abstract
Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs. Since human evaluation is costly, automated simulatability replaces human explainees with LLM simulators, as proposed in ConSim (Poché et al., 2025) for large-scale experiments. We qualitatively replicate and extend ConSim's ranking of explanation methods across the tested datasets, explanation families, and simulator LLMs, and identify two limitations. First, when class names are meaningful, simulators can obtain high simulatability by solving the classification task directly, without relying on the explanations. Second, class anonymization can reward explanations for leaking the hidden label mapping, a limitation we expose with a new classes-as-concepts baseline. These results are consistent with a shortcut hypothesis: in the tested settings, simulator predictions mainly rely on task priors, while explanations produce small changes. We derive recommendations for more robust automated simulatability evaluations.