Reinforcement learning improves calibration of reasoning model decisions
OpenJev-RLCD: A Working RLCD Implementation
Artificial IntelligenceComputation and Language
Summary
When AI models give answers with chances, those chances need to be accurate or 'calibrated' to be useful. The authors created a method called RLCD that helps reasoning models better estimate how confident they should be by learning from feedback on their reasoning steps. They tested this on two problem-solving tasks, and RLCD performed as well or better than existing methods, especially in knowing when it might be unsure. However, when uncertainty comes from people disagreeing on answers, RLCD cannot outperform classic methods. The code and results are available online.
What this means in practice
- •For machine learning engineers: Improve reasoning model confidence estimates to make better decisions in prediction tasks that require calibrated probabilities.
- •For ai system integrators: Deploy models that can better identify uncertain predictions and decide when to ask for further verification or human review.
Authors
Zhimin Gao, Pichao Wang
Abstract
Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive\% of the items at $\le$5\% error, versus \gvGrpoCovFive\% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.