SparsP speeds up video diffusion models by cutting GPU communication
SparSP: Exploiting Communication Sparsity for Sequence-Parallel Video DiTs
Distributed, Parallel, and Cluster ComputingMachine LearningPerformance
Summary
Video generation with Diffusion Transformers is powerful but very slow due to the heavy data exchange between GPUs. The authors found a way to make this communication smarter by sharing only the important parts that models need during their processing. They built a system called SparSP that arranges data and messages efficiently to reduce waiting and data traffic across GPUs. This system speeds up video generation by about 1.2 to 1.7 times while using less bandwidth, making diffusion models easier to run on common multi-GPU setups.
What this means in practice
- •For machine learning engineers: Run large video diffusion models faster on commodity multi-GPU servers by reducing GPU communication overhead with SparSP's sparse communication design.
- •For video production teams: Generate high-quality videos from AI models more efficiently, enabling quicker iteration and experimentation due to SparSP’s improved multi-GPU video diffusion inference.
Authors
Desen Sun, Xinrui Zhong, Yuke Wang, Sihang Liu
Abstract
Diffusion Transformers have become the dominant architecture for video generation. Their substantial computational cost motivates scaling inference across multi-GPU servers, yet efficient scaling remains challenging on commodity GPUs connected via PCIe, whose bandwidth is limited. Although sparse attention substantially reduces computation, its implications for communication remain underexplored. This paper argues that attention sparsity should be treated as a communication primitive. We present SparSP, an efficient sparse sequence parallel communication system that co-designs token distribution, communication routing, and asynchronous execution for sparse video diffusion models. First, Dependency-Aware Placement distributes sequence blocks according to diffusion models' sparse attention patterns. Second, Demand-Directed KV Routing transfers KV blocks directly to requesting GPUs without intermediate relays. Third, a Decoupled Transfer Runtime separates communication from GPU computation to reduce resource contention and maximize effective bandwidth. Our evaluation shows that SparSP improves attention performance by 1.38- 1.5$\times$, achieves an average 1.17$\times$ (up to 1.69$\times$) end-to-end speedup across three representative servers and three video diffusion models, and reduces communication volume by 12.54-23.05%. Moreover, we achieve an average 1.53-1.76$\times$ bandwidth improvement over NCCL.