Majority-vote pseudo-labels improve test-time reinforcement learning reliability
On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
Artificial Intelligence
Summary
People often try to teach AI models new tasks without giving them labeled examples, by letting the AI guess the answers itself and learn from those guesses. This paper looks at how reliable these self-generated guesses are when many guesses are combined by majority vote. The authors found that some errors come from differences in how prompts work and introduced a new method to make learning more cautious with less reliable guesses. Their method also keeps older knowledge from being forgotten. Tests showed their approach works better or as well as previous methods on several reasoning tasks.
What this means in practice
- •For ai developers: Improve online adaptation of language models to new domains without labeled data using more reliable pseudo-labeling methods.
- •For automated reasoning engineers: Enhance the robustness of reasoning models by reducing errors caused by varying prompt sensitivities during test-time learning.
Authors
Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming Gong
Abstract
Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.