CalibSFT improves confidence calibration in large reasoning models
On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models
Machine Learning
Summary
Large reasoning models often get too confident in their answers, even when they're unsure or wrong. The authors found that this happens because the models tend to stick to a narrow range of confidence levels, making it hard to train them to be better calibrated. They created a new method called CalibSFT that fine-tunes the model to better spread out its confidence levels before further training. This helps the models become more accurate at knowing when they might be right or wrong, improving how well their confidence matches their actual performance.
What this means in practice
- •For machine learning engineers: Improve model confidence accuracy to better handle uncertainty in AI-driven decision-making systems.
- •For ai service operators: Enhance selective prediction to decide when to trust model answers or defer for human review.
Authors
Shuoyuan Wang, Beier Luo, Hao Zeng, Chengyao Yu, Songxin Zhang, Zejian Xie, Bingyi Jing, Hongxin Wei
Abstract
Large reasoning models (LRMs) often suffer from overconfidence when expressing their uncertainty. Confidence-aware reinforcement learning (RL) offers a promising way to optimize calibration. However, it relies on on-policy rollouts and is thus constrained by the model's pre-RL confidence distribution, which we term confidence prior. In this work, we reveal that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL. Theoretically, we prove that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk. To overcome this exploration bottleneck, we propose CalibSFT, a plug-and-play supervised fine-tuning stage that shapes a calibrated confidence prior with broad support before RL. For each question, CalibSFT constructs confidence targets combining its success rate with response-level correctness, which provably preserves proper-scoring optimality, and then balances training responses across the confidence spectrum to enable diverse confidence exploration during RL. To learn from incorrect responses without imitating their reasoning, CalibSFT introduces correctness-conditional supervision, guiding confidence across all responses while supervising reasoning only on correct ones. Across 16 mathematical and general reasoning benchmarks, incorporating CalibSFT reduces calibration errors and improves discrimination across five representative RL algorithms while preserving comparable accuracy. Furthermore, CalibSFT delivers practical benefits for downstream selective prediction and model routing. Our code is available at https://github.com/ml-stat-Sustech/verbalized-confidence-training.