Language models often hide correct answers in their internal states
Legible Failures: Detecting and Repairing In-Context Binding Errors
Machine Learning
Summary
Sometimes, language models give wrong answers not because they lack information, but because they fail to use what they know. The authors show that a special tool called a probe can find the right answer inside the model's hidden layers, even when the model's output is wrong. They also demonstrate that using the probe’s findings can help detect when the model makes mistakes and even fix some errors without needing the correct answer upfront.
What this means in practice
- •For machine learning developers: Improve language model reliability by detecting and correcting internal binding errors using probe-guided interventions without requiring correct labels.
- •For natural language processing engineers: Use probe outputs to identify when language models misunderstand entity relationships within prompts to enhance downstream task accuracy.
Authors
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. Hari
Abstract
A wrong answer does not show whether the model lacked the needed information or held it and failed to use it. On an entity-obligation binding task, a language model can emit an incorrect prompt-supplied binding while a linear probe can recover the correct one from its frozen hidden state. We measure how often this occurs across 16 public checkpoints, each evaluated with three seeds. We fit a probe on a training fold, select its layer on a validation fold, and report results on a disjoint test fold. On the trials each model gets wrong, probe accuracy exceeds the strict present-obligation baseline, 1/K = 0.125, by +0.196 (95% CI [+0.101, +0.296], bootstrapped over models). A query-entity counterfactual rules out token presence and recency. A score built from the sign of probe-output disagreement improves failure detection over the model's own confidence by +0.079 AUROC (95% CI [+0.036, +0.126]). Raw probe confidence gives no measurable improvement over model confidence. Steering the residual stream toward the probe-decoded binding, with no gold label, raises accuracy on all eight models tested by a mean of +0.168 (95% CI [+0.066, +0.280]). Where recent studies report that probe-detected errors are resistant to interventions, we find that in-context binding is a setting in which probes are actionable.