FourSight improves RL model accuracy with stable low-bit deployment

FourSight: Stable FP4 RL Post-Training via Sequence Filtering and Sparse Channel Compensation

Artificial IntelligenceMachine Learning

Summary

Reinforcement learning models often lose performance when their computations are simplified to use very small numbers to save space and speed. The authors found that most errors come from a few unusual sequences and a limited number of channels rather than being spread evenly. They created FourSight, which filters out these problematic sequences during training and uses more precise numbers only in important channels. This approach helps models run faster with low-bit numbers while keeping their quality almost as good as more precise formats.

What this means in practice

Authors

Yining Wei, Tianyuan Wu, Gu Gong, Yuechen Tao, Yinghao Yu, Xiaowen Chu, Wei Wang

Abstract

Reinforcement learning (RL) post-training increasingly targets low-bit deployment, with NVFP4 weight-and activation quantization (W4A4) becoming the state-of-the-practice format. However, integrating NVFP4 into RL degrades rollout quality and amplifies cross-engine training-inference mismatch (TIM), destabilizing training. We show that these failures are concentrated rather than uniform - severely mismatched tail tokens concentrate in a few outlier sequences, while policy weights remain largely intact and quantization error concentrates in a few high-variance activation channels. Guided by this, we present FourSight, an algorithm-system co-design with two targeted solutions: TIM-aware sequence filtering that excludes drifted trajectories from policy updates, and sparse precision compensation that keeps only top-variance channels in BF16 while retaining native NVFP4 elsewhere. We further design a fused SpliceGEMM kernel with specialized prefill and decode datapaths to minimize overhead. FourSight sustains near-BF16 accuracy and strong FP4 deployment quality across models and engines, while improving step-to-accuracy by 1.76-2.60x and staying within 2.3% of FP4 step time up to 120B-scale models.