Confidence scores adapt to reasoning changes in reinforced language models
Test-Time Confidence Adaptation under RL-Induced Trajectory Shift
Artificial Intelligence
Summary
This paper looks at how large language models give confidence scores about their answers. When these models learn new strategies through reinforcement learning, their way of thinking changes and confidence scores can become less accurate. The authors found that even with these changes, answers based on more stable reasoning steps still tend to be better. They created a new method that uses this idea to adjust confidence scores without needing extra labels, improving reliability across different tasks and models.
What this means in practice
- •For ai system developers: Improve confidence estimation for language models adapting through reinforcement learning to maintain answer reliability.
- •For qa platform engineers: Enhance answer confidence scoring in question-answering services when models update or change their reasoning style.
Authors
Zhenzhen Ren, Xiaoqing Zhang, GuoBiao Li, Bowen Fang, Yanxi Wang, Xinpeng Zhang, Ke Han
Abstract
Trajectory-based confidence estimation has shown promising performance for large language models (LLMs) by leveraging rich signals from their reasoning processes. However, we find that such confidence estimators can degrade substantially after agentic reinforcement learning (Agentic RL) as it induces substantial shifts in the reasoning trajectories. More importantly, despite these shifts, we uncover a reliability structure that is largely preserved across Base and RL models: $\textbf{more}$ $\textbf{stable}$ $\textbf{trajectories}$ $\textbf{remain}$ $\textbf{more}$ $\textbf{likely}$ $\textbf{to}$ $\textbf{produce}$ $\textbf{correct}$ $\textbf{answers}$. This preserved relationship provides a transferable, label-free reliability signal for post-RL confidence adaptation. Building on this finding, we propose $\textbf{StaR-TTA}$, a stability-guided framework for label-free test-time confidence adaptation. StaR-TTA jointly learns from correctness supervision and stability-derived reliability supervision on labeled Base trajectories, establishing the connection between trajectory stability and answer reliability. At test time, it reuses stability-induced reliability rankings from unlabeled RL trajectories to adapt confidence scores to the shifted trajectory distribution, without requiring target correctness labels. Across four QA benchmarks, StaR-TTA achieves an average AUROC of \(0.721\), with consistent improvements across unseen datasets, model families and scales, and different Agentic RL algorithms.