Language model confidence reports often unreliable and sensitive
An Analysis of Training-Free Self-Reported Confidence in Language Models
Computation and Language
Summary
Large language models can say how confident they are in their answers, but it’s unclear if these confidence numbers truly reflect accuracy. The authors studied several ways models express confidence without extra training and found that simply asking the model to verbalize confidence works surprisingly well at spotting correct answers. However, confidence can change a bit depending on how the question is asked, and sometimes confident answers are still wrong. This means that while models’ self-reports can help, they aren’t fully reliable and can be influenced by repeated mistakes and noisy tests.
What this means in practice
- •For natural language processing engineers: Use direct verbalized confidence from language models to better assess answer reliability during model output evaluation.
- •For customer support platform developers: Integrate language model confidence scores cautiously, recognizing variability in self-reported confidence and correlated errors when automating responses.
Authors
Lukas Meyer, Sofia Rossi, Wei Chen, Thomas Laurent, Yiming Li
Abstract
Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated rhetoric. We analyze three training-free signals: confidence verbalized with the answer, post-hoc $P(\mathrm{True})$, and agreement with three additional generations on the same 100 TriviaQA questions for two model families. Direct verbalization is a surprisingly strong baseline: after auditing benchmark errors, it reaches AUROC 0.956 and 0.937 for correctness prediction. Three-sample agreement is substantially weaker (0.765 and 0.790), and a fixed interpolation with verbalized confidence has no statistically reliable benefit. Four of nine errors from one model and two of eight from the other receive unanimous sample support, showing that self-consistency can amplify shared misconceptions. Re-eliciting confidence for the same fixed answers with equivalent prompts changes scores by 0.043 to 0.084 on average and flips 4\% to 9\% of decisions at a 0.8 threshold. An exploratory audit of 100 confidence-tagged biography claims further finds only a modest confidence gap between supported and contradicted claims. These results argue that useful self-reports remain sensitive to elicitation, correlated errors, and benchmark noise.