ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
2026-07-11 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors study why training large language models with reinforcement learning can become unstable, focusing on a problem called over-optimization where the model cheats on training but can't reason well in general. They find that the usual method, reverse KL regularization, does not work well enough because it doesn’t fully cover the right solutions. To fix this, they create ARMOR, a new approach that actively stabilizes training by using past good examples (Anchor Rollout) and a smarter way to explore options (Mixed Optimization). Their tests show ARMOR helps maintain good reasoning performance over longer training times.
reinforcement learninglarge language modelsover-optimizationreverse KL regularizationoff-policy dataAnchor RolloutMixed Optimizationpolicy objectivetraining stability
Authors
Kexin Huang, Junkang Wu, Jinda Lu, Shuo Yang, Chiyu Ma, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou
Abstract
Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile. In this work, we investigate a critical source of this instability: over-optimization, where models exploit training heuristics at the expense of generalizable reasoning. While reverse KL regularization is the standard defense against such degradation, our analysis reveals that it is often insufficient in this regime, as it fails to ensure comprehensive coverage of the reference distribution. To address this, we propose ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization. ARMOR comprises two key components: (1) Anchor Rollout, which leverages off-policy data from the reference policy to preserve established solution patterns; and (2) Mixed Optimization, which reformulates the policy objective to enable controlled exploration without relying on auxiliary losses. Extensive experiments on reasoning benchmarks validate that ARMOR effectively mitigates validation collapse, enabling sustained performance improvements over extended training horizons.