Confidence alignment improves reinforcement learning in language models
SCARL: Sequence Confidence-Aligned Reinforcement Learning
Machine LearningArtificial Intelligence
Summary
Language models learn by getting rewards when their answers pass a test, but sometimes they are confident about wrong answers or uncertain about right ones. The authors noticed this mismatch between how confident the model feels and how good its answer actually is. They created a new method called SCARL that adjusts the learning process so the model's confidence better matches the correctness of its answers. This helps the model explore more and improves its performance on reasoning tasks.
What this means in practice
- •For ai product developers: Improve language model reasoning accuracy and confidence calibration in natural language generation services by integrating confidence-aligned reinforcement learning.$Commercial implications: Enables more reliable AI-generated content for product features requiring high accuracy and user trust.
- •For machine learning engineers: Enhance training of reinforcement learning policies for language models by reducing entropy collapse and improving exploration with a confidence-alignment method.
Authors
Zhenghong Ke, Jingbo Zhou, Hua Wu
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning by scoring sampled responses with a rule-based verifier and reinforcing those that pass. Where a classifier's training couples predicted probability to labels, nothing ties an RLVR policy's confidence in a response to its reward: aligned training would drive responses with the same advantage toward equal confidence, yet the advantage is blind to confidence. Under this sequence-confidence misalignment, a success the policy barely found earns the same update as one it produces with near certainty, and a confident error is corrected no harder than an exploratory one. We propose SCARL (Sequence Confidence-Aligned Reinforcement Learning), which reshapes, but never re-signs, each advantage with the alignment gap between the response's group-relative confidence and the outcome its advantage certifies, normalized into shares of each class's advantage budget. Misaligned samples absorb the force, responses with the same advantage are driven toward equal confidence, and repairing the misalignment doubles as a remedy for the entropy collapse that erodes exploration. SCARL provably reduces to its base estimator when a class's confidences agree, adds no extra passes and one hyperparameter. On Qwen3-4B-Base it lifts average Pass@32 by 2.4 points over DAPO and is best among nine compute-matched methods on both coverage and per-draw accuracy; applied unchanged across four estimators and four backbones, it improves 14 of 16 settings while preserving policy entropy and widening the confidence-correctness separation. Our code is available at https://github.com/kezhh163/scarl.