CadenceRL improves reinforcement learning rollout speed on mixed hardware

Reshaping Rollout Workloads for Asynchronous RL Post-Training on Heterogeneous Accelerators

Distributed, Parallel, and Cluster Computing

Summary

Reinforcement learning often needs to run many long tests, or rollouts, that are slow and use different types of computer hardware. The authors show that balancing the workload better across different machines can make these rollouts faster and more efficient. They created CadenceRL, a method that reshapes workloads by breaking long tasks into shorter ones and sending the hardest parts to the fastest machines. This approach improves overall processing speed and reduces delays without needing manual configuration.

What this means in practice

  • For machine learning engineers: Speed up reinforcement learning post-training tasks by automatically balancing rollout workloads across different hardware accelerators without manual tuning.
  • For cloud infrastructure managers: Optimize use of mixed hardware clusters for reinforcement learning jobs, getting better throughput and lower latency by workload reshaping with CadenceRL.

Authors

Jiahui Li, Hao Nie, Yibo Zhu, Pengjin Xie, Yu Zhou, Xiaolong Zheng, Liang Liu, Huadong Ma

Abstract

Reinforcement learning (RL) post-training increasingly relies on long-horizon, multi-turn rollouts. As post-training jobs outgrow a single cluster, rollout pools assembled across clusters introduce hardware heterogeneity. Rollout scheduling must serve two stakeholders: the hardware needs high aggregate decode throughput, while each trajectory needs to finish quickly. The tension arises from the memory-bandwidth-bound nature of autoregressive decoding. A large active batch amortizes weight reads for high throughput but leaves each trajectory a smaller bandwidth share and a longer completion time. The scheduling objective is therefore specialization, letting different workers serve different roles. Heterogeneous hardware further enables this specialization. High-bandwidth accelerators favor long-context work, while cost-efficient accelerators sustain large batches. Workload evolution makes this specialization difficult to sustain, and dynamic reassignment faces a circular dependency because a move's benefit depends on subsequent placement decisions. We present CadenceRL, which bypasses this dependency through structural workload reshaping rather than per-move benefit estimation. Pacing replaces long-context trajectories with shorter ones, providing a structurally positive transformation that sustains large active batches for high throughput. When accumulated staleness demands faster completion, concentration directs the residual long-context tail onto high-affinity workers. Late-bound KV preparation stages accumulated prefixes before a destination is selected. On heterogeneous rollout pools, CadenceRL improves decode throughput by up to 48% and reduces P95 trajectory latency by up to 64%. Adding high-bandwidth accelerators reduces tail latency, while adding cost-efficient accelerators increases throughput, without manual routing configuration.