MoE language models gain accuracy with symmetric FP4 reinforcement learning

Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

Machine LearningArtificial Intelligence

Summary

When training certain big language models called Mixture-of-Experts (MoE), using low-precision numbers makes the models less accurate. The authors found that a recovery method that helps models with regular structures does not work for MoE models because MoE has a special way of choosing which expert to use. They discovered that making the correction method symmetric between training and using the model fixes the problem and improves accuracy. Their new method, MoeResQ, helps MoE models get closer to the performance of full-precision models.

What this means in practice

  • For machine learning engineers: Improve the accuracy of MoE large language models running in low-precision FP4 by applying symmetric recovery techniques during reinforcement learning fine-tuning.
  • For cloud ai infrastructure teams: Deploy more efficient MoE language models that maintain accuracy despite aggressive quantization, enabling lower memory and computation costs in production.

Authors

Hei Yi Mak, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Huan Lin, Zhengzhi Tang, Zhexu Liu, Yao Wang, Yaoyuan Wang, Anandharaju Durai Raju, Tanzila Rahman

Abstract

In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoeResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoeResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.