Diagnosing Correctness Probes under Self-Judgement Confounding
Computation and LanguageMachine Learning
Summary
The authors studied how language models decide if their answers are correct, comparing the model's own judgment (SJ) with actual correctness (OC). They created tricky cases where these two disagree and found that models tend to trust their own judgment over true correctness, especially in complex tasks like math and factual questions. This pattern appears in deeper layers of the model and happens consistently across different models and tests. They conclude that just because a model's internal signals transfer well, it doesn't mean those signals represent true objective correctness.
Authors
Yi-Long Lu
Abstract
Hidden-state readouts can predict whether language-model outputs are correct, but objective correctness (OC) usually agrees with the model's own self-judgement (SJ), leaving the decoded signal semantically ambiguous. We construct conflict cases in which OC and SJ predict opposite readout orderings. On high-confidence disagreements, conventional correctness-labelled contrasts often rank incorrect/self-endorsed responses above correct/self-rejected responses, following SJ rather than OC. We estimate factorial SJ- and OC-associated directions and evaluate their polarity across mathematical reasoning and factual recall. Across four instruction-tuned models up to 14B parameters, the SJ-associated direction transfers above chance in both cross-domain directions for every model, whereas the OC-associated direction has a below-chance point estimate for the expected OC ordering in every corresponding condition. This transfer asymmetry develops across middle-to-late layers, persists under answer-likelihood, sequence-length, and null-direction controls, and extends to MMLU and binary TruthfulQA without target-domain direction fitting. Across the studied models and diagnostic subsets, the most reliably transferable component preserves SJ-associated polarity. Transferability alone therefore does not establish objective-correctness semantics.