Mechanistic interpretability faces challenges from minor input changes
The Misery of Mechanistic Interpretability: A Formal Perspective
Machine LearningArtificial Intelligence
Summary
Understanding how large language models work inside is hard because they are like black boxes. To make sense of them, people train smaller, simpler networks that highlight important features people can understand. The authors show that small changes in input can confuse these explanation networks, making their interpretations unreliable. They introduce a way to formally check how trustworthy these explanations are, even against tricky inputs. By training these explanation networks with this checking method, they make interpretations more stable and trustworthy.
What this means in practice
- •For ai safety auditors: Verify and improve explanations of language model decisions to support safety assessments under varied input conditions.
- •For machine learning engineers: Use verification-aware training to develop more reliable interpretability tools for debugging and model analysis.
Authors
Tobias Ladner, Matthias Althoff
Abstract
Mechanistic interpretability has become the dominant lens for understanding frontier language models, as their inner workings are complex and inherently black boxes. To gain insights into these models, interpretable replacement networks (IRNs) are trained at all layers, exposing interpretable features through sparsely activated neurons. However, the faithfulness of an IRN is usually evaluated only empirically on clean data, and we show that even semantically minor input perturbations flip the dominant IRN features-and thus the human-understandable interpretation-across five open-weight model families (GPT-2 small, Gemma 2 2B, Gemma 3 1B, Llama 3.2 1B, R1-Distill-Qwen 1.5B). We propose the first formal verification framework for the faithfulness of an IRN, where reachability analysis certifies a sound upper bound of the faithfulness gap in adversarial scenarios. Moreover, we show that verification-aware training of IRNs substantially tightens this certified bound, restoring a feature-level interpretation that safety auditors can act on. Together, these results give, to the best of our knowledge, the first formal guarantees for mechanistic interpretability of large language models.