Answer correctness signals appear in multiple places inside language models
Locating Answer-Correctness Signals in Frozen Large Language Models
Artificial Intelligence
Summary
Determining if a language model’s answer is correct is important but challenging, especially when the model uses outside information. The authors studied where inside a frozen language model correct answers show up during a single run without extra checks. They found that signals of correctness are mainly in the part of the answer itself and come from different internal data types. Combining these signals gives the best clues, especially when the model faces new or tricky information. This insight helps design systems that better judge if the model’s answer is trustworthy.
What this means in practice
- •For natural language processing engineers: Build systems that detect correct answers from a single pass inside frozen language models to improve answer validation without extra runs.
- •For information retrieval developers: Develop retrieval controllers that use fused internal signals to gate when retrieved evidence contributes to trustworthy model answers.
- •For chatbot product teams: Create user-facing tools that better assess answer reliability using combined internal signals from language models without costly repeated queries.$Commercial implications: Enables chatbots to confidently validate answers in real time, improving user trust and reducing server costs, which supports selling more reliable conversational products.
Authors
Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung
Abstract
Language models expose internal signals that predict whether an answer is correct, readable from a single forward pass of a frozen model without additional generations. Yet existing probes often commit to one signal family or layer and can be brittle under distribution shift; in retrieval-augmented settings, many specialized detectors instead target passage faithfulness, which can diverge from correctness when retrieved evidence is unhelpful or conflicting. We therefore ask where answer correctness is readable, which internal signal families carry it, and how they should be combined. We search over hidden states, token probabilities, residual-stream features, attention, and their fusion, treating the selected readouts as a predictive measurement rather than a mechanistic localization. We run this analysis separately in closed-book and with-context settings, since context can change which readouts are informative. A consistent anatomy emerges: correctness concentrates in the answer span, recovered from the answer tokens even under retrieval, and the families carry it complementarily, so fusing them helps most out of distribution, where a single signal is weakest. The protocol is effective across two backbones and gates a retrieval controller as one downstream use.