KV4-RL speeds up reinforcement learning with efficient cache quantization
KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
Machine LearningArtificial Intelligence
Summary
Reinforcement learning often slows down when generating data because many processing steps take too long. The paper shows that compressing key memory (KV cache) with low-bit numbers speeds up this step much more than other types of compression. However, too much compression can hurt learning accuracy. The authors introduce KV4-RL, which uses mostly low-bit compression but keeps important recent information less compressed, balancing speed and learning quality. Their method achieves nearly twice as fast data generation while maintaining accuracy close to full-precision training.
What this means in practice
- •For machine learning engineers: Accelerate reinforcement learning training by compressing key-value cache without significant accuracy loss.
- •For ai infrastructure teams: Deploy faster, lower-precision memory formats for transformers in asynchronous rollout systems to reduce GPU compute time.
Authors
Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang
Abstract
Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step. We characterize this \emph{KV concentration} and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons. However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS). We propose \emph{KV4-RL}, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. Protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves $1.80\times$ cumulative rollout speedup and $1.27\times$ cumulative full-iteration speedup over BF16 rollout, reaching $1.97\times$ rollout speedup over the final 50 iterations. Under a common BF16 evaluator, its final six-benchmark mean accuracy is within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.