Verbalized confidence overtakes log-probs in AI judging post 2025 models
Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
Computation and Language
Summary
Many AI models used to score their answers by looking at log-probabilities, but newer models made after 2025 work better if they say how confident they feel in words. The authors find that this 'verbalized confidence' is more reliable and handles tricky, subjective questions better. They also show that adding ways to warn about overconfidence and to let the model debate itself improves these scores without hurting accuracy in newer models. This means using the model's own words about its confidence is a better way to judge answers now than the old methods.
What this means in practice
- •For ai system evaluators: Use verbalized confidence to improve judgment accuracy and robustness of AI evaluations on post-2025 language models.
- •For natural language processing engineers: Incorporate overconfidence warnings and self-debate to enhance reliability and calibration in model scoring for subjective NLP tasks.
Authors
Yu-Chung Hsiao
Abstract
Verbalized confidence, long dismissed as overconfident, coarse, and prone to round-number clustering, is now the more robust soft-scoring mechanism for LLM-as-a-Judge on top-tier proprietary models. Across SummEval, AggreFact, and HelpSteer2, spanning up to 18 LLMs, we show that the standard advice to prefer log-probabilities no longer holds on post-2025 models, where verbalized confidence is the better signal. We call this a compatibility shift. On top of a standard verbalized-confidence baseline, we introduce two new ingredients: an overconfidence advisory and self-debate. Together they improve calibration, score-distribution spread, and robustness to task subjectivity. We further observe a generation effect: post-2025 models accommodate these two additions with little balanced-accuracy cost, whereas pre-2025 models pay a measurable penalty. Compared with logprob-based G-Eval, verbalized confidence is the more subjectivity-robust soft signal on GPT-family top-tier releases. The shift is invisible under accuracy-only reporting. Rather than defaulting to hard predictions, we recommend broader use of soft scoring in LLM-as-a-Judge. More broadly, verbalized confidence has moved from a weaker substitute for logprobs to a practical soft-scoring mechanism for contemporary LLM judges.