Confidence signals vary widely across large language models and tasks
Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Knowing when a language model is sure about its answer matters to use it safely. The authors compared three ways to measure confidence in large language models across different tasks like fact-finding and math problems. They found that early signals about confidence can come from where the model focuses in its layers, but some signals are unstable or misleading. For example, the model’s verbal expression of confidence changes a lot with small changes in the prompt, and common error measurements don’t always show the true quality of confidence. This means different methods for measuring confidence can’t be used interchangeably without care.
What this means in practice
- •For software engineers: Improve the reliability of AI assistants by selecting better confidence measures to decide when the model should abstain from answering.
- •For machine learning ops teams: Develop more robust monitoring tools by understanding which confidence signals actually reflect model uncertainty in different tasks.
Authors
Debojeet Ghosh, Shreyansh Kumar, Ayushman Garg
Abstract
Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain. Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable. We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH). First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%). The pre-registered five-layer test supported this only marginally; a finer 21-layer follow-up, designed after those results, supports it more clearly. In most retrieval cells, however, a prompt-only TF-IDF classifier matches the probe, so the early signal may reflect surface features of the question. Second, verbalized confidence is unstable: slightly different prompt templates yield scores that barely correlate. Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution. Of five pre-registered hypotheses, two were falsified, one was partly supported, one was not supported, and one could not be tested because the base model's verbal confidence was degenerate.