Calibrating large reasoning models to reduce overconfidence errors
RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Language models trained to solve reasoning problems can become overconfident and give incorrect answers without signaling uncertainty. The authors propose a new training method called RL-ARC that helps models better estimate how confident they should be in their answers by incorporating how confident the model is in its reasoning process. This helps the model avoid being too sure about wrong answers and improves reliability without hurting its ability to reason. Their tests show the approach works well both on familiar and new types of questions.
What this means in practice
- •For machine learning engineers: Train language models that provide more reliable confidence scores alongside reasoning outputs in real-world applications with varying input types.
- •For ai product teams: Build AI systems that better recognize when their reasoning or answers might be wrong, improving trust and safety in deployments.
Authors
Gukhyeon Lee, SangKeun Lee
Abstract
Language models (LMs) are commonly trained with Reinforcement Learning with Verifiable Rewards (RLVR) to enhance their reasoning capabilities. However, since RLVR does not explicitly account for calibration during training, it can lead to severe calibration degradation, including overconfidence. Recent calibration-aware training methods for LMs, which incorporate objectives for uncertainty estimation into training, improve calibration but still exhibit overconfidence under distribution shift, while sacrificing reasoning performance. To this end, we propose RL-ARC, a calibration-aware training framework that jointly leverages reasoning confidence and answer confidence. Specifically, RL-ARC leverages reasoning confidence as an auxiliary signal for calibrating answer confidence, applying it as reasoning-guided regularization for correct cases and as an overconfidence penalty for incorrect cases. Comprehensive results across ID and OOD settings show that, beyond improving calibration, RL-ARC enables reasoning models to adaptively estimate confidence based on the given question without substantially sacrificing reasoning performance, thereby highlighting the importance of reasoning confidence for training reliable reasoning models.