Calibrate tools measuring large language models for reliable insights
Calibrating Interpretability Instruments Before Trusting Their Verdicts
Machine LearningArtificial IntelligenceComputation and Language
Summary
Measuring how big language models work internally is tricky because the usual tools can give convincing but wrong answers. The authors show that some common methods produce misleading results in specific ways that are hard to spot without careful checks. They describe six examples of these hidden errors and provide ways to detect and avoid them. Their approach helps make sure that claims about what the models learn inside are based on trustworthy measurements.
What this means in practice
- •For machine learning engineers: Check if interpretability methods yield trustworthy results during model debugging and refinement.
- •For model auditing teams: Detect measurement artifacts when assessing language model behaviors and decision processes.
Authors
Orion Reblitz-Richardson
Abstract
Causal claims about large language model (LLM) internals rest on measurements. Those might include a projection, a cosine, an ablation delta, or an interchange patch among others. These measurements fail in specific, diagnosable ways that return a plausible number instead of an error, so a broken instrument can easily read as a finding. A covariance-matched null can saturate until every direction looks typical, a per-head attribution can overshoot the true residual write threefold on reordered-normalization architectures, an interchange patch can go sign-chaotic because its outcome is pinned at a ceiling, or a read-from verdict can be an artifact of measuring past the layer where the model already decided. This note documents six such failures from a causal interpretability program on refusal and moral representation, spanning several papers and a four-model open-weight panel; each mode is established on one or two of the four. For each we give the tell that catches it and a protocol keyed to a detectable trigger (reordered normalization, massive activations, a low-dimensional decision channel), so we and readers can check whether a given setup is exposed. The discipline reduces to four moves: calibrate against a positive-control ladder, certify with an orthogonal cell, compute power before spending compute, and state every read-from verdict at a depth referenced to the model's commitment. The evidence is four architectures across three families within a single program; external replication across programs is future work.