Robust dual-channel optimizer improves policy learning with noisy rewards
Dual-Channel Robust Group-Relative Policy Optimization via Advantage and Sequence-Weight Estimation
Artificial Intelligence
Summary
Sometimes when teaching computers to make decisions, certain strange or extreme results can confuse them and cause problems. The authors developed a new method that separately handles unusual rewards and sequence evaluations to keep the learning process stable. This method improves the computer’s ability to learn from tricky data, making it better at tasks like math reasoning and summarizing long documents. Experiments show their approach works better than previous methods, especially when the data is noisy or contains odd outliers.
What this means in practice
- •For machine learning engineers: Improve training stability for reinforcement learning models handling noisy or extreme reward signals.
- •For natural language processing developers: Enhance summarization systems that process long texts by reducing sensitivity to rare or misleading token-level errors.
Authors
Zhongyi Li, Wan Tian, Xiang Xu, Yutian Xiao, Yikun Ban, Yijie Peng, Fuzhen Zhuang
Abstract
Group-relative policy optimization relies on reward-derived advantages and sequence-level likelihood weights, both of which can be sensitive to localized outliers. Extreme rewards can collapse the contrast among clean responses after group normalization, while token-level log-ratio perturbations can alter sequence weights and clipping decisions. We introduce RoVR-GSPO, a dual-channel robust optimizer that addresses these failure modes separately. Its reward channel combines robust reference estimation with bounded residual credit, while its ratio channel uses differentiable SoftRoVR aggregation to construct robust sequence weights. We provide stability and efficiency analyses for both channels. Experiments on mathematical reasoning, long-context summarization, and tool-call annotation show consistent improvements over GSPO, while controlled perturbation studies demonstrate stronger robustness to reward contamination and token-ratio anomalies.