Fast video generation improves quality with smarter attention scheduling
iCATS: Fast Video Generation via Interaction-Aware Sparse Attention and Timestep-Adaptive Sparsity
Computer Vision and Pattern Recognition
Summary
Generating videos using AI can be slow and sometimes less accurate if the system tries to speed up by focusing only on important parts. The authors found a better way to guess which parts matter by looking at how video bits interact with each other, making the process more precise. They also let the system decide how much to simplify at different steps to avoid mistakes. Additionally, they improved how computers handle the work to run faster without losing quality.
What this means in practice
- •For video production teams: Create high-quality AI-generated videos faster by using adaptive attention strategies that reduce computation while preserving detail.
- •For graphics hardware engineers: Optimize GPU kernel designs to improve efficiency when running sparse attention computations with irregular cluster sizes during video AI tasks.
Authors
Chengfeng Han, Baole Ai, Xianlu Bian, Jie Yao, Zilong Huang, Ang Wang, Dandan Ding
Abstract
Training-free sparse attention offers a practical acceleration solution to Diffusion Transformers (DiTs) via reducing computations without fine-tuning. It typically involves estimating the importance of query-key regions and deriving sparse masks to compute only the important candidates, which inevitably introduces approximation errors that may degrade generation quality. To better balance the efficiency-quality trade-off, we propose iCATS, integrating improved importance estimation and sparse mask construction with an efficient hardware execution strategy. Specifically, for importance estimation, unlike previous works that perform independent clustering over query and key tokens based on feature similarity to estimate attention scores, iCATS demonstrates that clustering based on query-key dot-product interactions is more accurate and further reformulates this objective as a simple quadratic form for low-cost computation. For sparse mask construction, instead of using a fixed top-p rule, we observe that tolerance to sparse approximation errors varies across denoising timesteps and therefore introduce an SNR-guided sparsity schedule to adjust sparsity dynamically, leading to higher accuracy. Finally, for hardware execution, we devise a tail-merging strategy to reduce padding overhead caused by irregular cluster sizes, improving GPU kernel utilization. Extensive experiments show that iCATS achieves $2.03\times$ acceleration with 31.017 dB PSNR on HunyuanVideo-T2V-13B and $1.55\times$ acceleration with 29.301 dB PSNR on Wan2.1-T2V-14B, delivering a state-of-the-art efficiency-quality trade-off.