Router drift control improves expert routing in reinforcement learning
Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning
Machine LearningArtificial Intelligence
Summary
Reinforcement learning systems sometimes train with data that doesn’t exactly match how they operate when making decisions, causing performance problems. This issue is larger in models called Mixture-of-Experts, where decisions about which expert to use can drift and degrade results. The authors show that a previous fix can bias learning and propose Router Drift Control, which detects when decisions disagree and filters out problematic examples. Their method keeps learning signals intact and works better as the difference increases, especially in asynchronous setups.
What this means in practice
- •For reinforcement learning engineers: Reduce performance degradation in MoE RL models by managing routing mismatches during training and inference.
- •For distributed ai system architects: Improve training stability and accuracy in asynchronous reinforcement learning deployments by integrating router drift detection.
Authors
Liangyao Jin, Zhixin Wang, Liming Liu, Mingjun Zhang, Zhengzhou Zhu, Fangcheng Fu, Jinlong Hou, Yuan Cheng, Xiaohe Hu, Ping Zhang, Siyuan Feng
Abstract
Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training.The resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout--training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.