FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
2026-07-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how a special type of attention method, used in video-generating AI models called Video Diffusion Transformers, can be made faster when running on multiple GPUs. They found that uneven work distribution among GPUs was slowing things down, so they built a system called method that balances the work better without extra training. Their system smartly manages which parts of the video to pay attention to, moves heavy tasks between GPUs, and fills in idle times for efficiency. Tests showed that method speeds up attention calculations by over 4 times and doubles inference speed while keeping video quality good.
Video Diffusion TransformersSelf-attentionSparse attentionMulti-GPU sequence parallelismTop-p routingLoad balancingFlashAttentionStep-distilled Wan2.2 I2VInference speedupSpatio-temporal sequences
Authors
Hao Liu, Chenghuan Huang, Ye Huang, Zhiying Wen, Hao Liu, Mohan Zhang, Chen Li, Ziyang Ma, Jing Lyu, Jiangsu Du
Abstract
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-$p$ routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present \method{}, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. \method{} uses Top-$p$ routing, a Top-$k$ safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, \method{} reduces average load imbalance from 1.34 to 1.08 and delivers a $4.41\times$ attention speedup over FlashAttention, while achieving a $2.02$--$2.11\times$ DiT inference speedup with competitive video quality.