TTPO: Test-Time Policy Optimization
2026-08-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors noticed that current methods for improving math reasoning in language models need correct answers, which makes training during testing hard. They found that when models disagree with a majority vote, they usually make mistakes anyway. Based on this, they created a new method called Test-Time Policy Optimization (TTPO) that learns by focusing more on agreeing answers and penalizing confident mistakes, even without true labels. Their approach works well on challenging tests and helps models improve while testing without needing extra labels.
Post-training methodsReinforcement LearningOn-Policy Self-DistillationTest-Time TrainingPseudo-labelsMajority VoteMathematical ReasoningLanguage ModelsPolicy Optimization
Authors
Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen
Abstract
Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.