StageBreaker speeds up training for video reinforcement learning models

StageBreaker: Shifting Video RL Post-Training Scheduling from Stages down to Data Dependencies

Distributed, Parallel, and Cluster Computing

Summary

Training video models using reinforcement learning can take a long time because of how tasks are scheduled and how GPUs are used. The authors propose a new way called StageBreaker that looks at the fine details of data flows instead of broad stages, allowing different parts of the training to run at the same time. This approach reduces waiting times, avoids doing some steps twice, and balances the work across GPUs better. Their method made training up to 41% faster compared to traditional scheduling.

What this means in practice

  • For machine learning engineers: Accelerate training pipelines for autoregressive video models by using StageBreaker to reduce computation bottlenecks and improve GPU utilization.
  • For high-performance computing teams: Optimize GPU scheduling for reinforcement learning workflows that involve complex reward calculations and long video sequences.

Authors

Li Peng, Chenghuan Huang, Ziyang Ma, Jing Lyu, Xiang Chen, Jie Zhang

Abstract

Reinforcement learning (RL) post-training has emerged as the key paradigm for improving autoregressive video models. Emerging vision-language reward models further strengthen post-training with accurate rewards, but their long chain-of-thought inference substantially raises the associated costs, while the pursuit of long videos prolongs policy training. Despite these rising costs, existing video RL systems still adopt coarse-grained stage-centric execution, which stalls policy training until all reward evaluation finishes, replays every trajectory twice due to a further barrier within policy training, and induces load imbalance across GPUs. To address the challenges, we propose StageBreaker, a fine-grained data-dependency-driven scheduling framework, which schedules the true dataflow of an RL step rather than its stages. Specifically, StageBreaker overlaps policy training with reward evaluation by exploiting reward-separable policy training, and merges the two trajectory replays into one by leveraging loss-independent back-propagation. It further balances GPU load by scheduling computation at the finer granularity. Our evaluation results show that StageBreaker achieves a speedup of up to 1.41$\times$ over stage-centric execution.