ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
2026-07-27 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors explain that training large language models using reinforcement learning is unstable because training and actual use (inference) are done differently, especially in terms of precision and system design. They introduce Adaptive Control Reinforcement Learning (ACRL), a method that keeps these differences under control to make training more stable. This approach also helps the model explore better and improves its accuracy. Their tests show that ACRL works well even when using lower precision during inference and outperforms some existing methods.
Reinforcement LearningLarge Language ModelsTraining-Inference DiscrepancyLow-Precision QuantizationFP8 QuantizationPolicy EntropyAdaptive ControlImportance SamplingBF16 Precision
Authors
Wenwu Fan, Qihong Lin, Zhijie Xia, Zhuo Zheng, Sihao Wang, Qiang Chen, Liangsheng Zhu
Abstract
Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.