Training free allocation boosts video diffusion transformer attention quality
Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
Video diffusion transformers use a process called attention to analyze video data, but doing this fully is very slow. The authors found that paying attention (sparsity allocation) differently across parts and times in the model affects video quality a lot, and the important parts stay the same for different videos. They created ALLOT, a method that decides how to distribute this attention without needing extra training by testing each part's impact on video quality. This approach speeds up video processing while improving quality compared to other methods.
What this means in practice
- •For video processing engineers: Improve real-time video generation quality and speed by optimizing attention allocation without retraining models.
- •For machine learning infrastructure teams: Reduce computational resources needed for video diffusion transformer deployments by efficient attention sparsity management.
Authors
Tianze Xia, Xiaoling Zhou, Ruya Lan, Yiqiao Liao, Chengru Song, Shiming Xiang
Abstract
Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. However, existing methods either overlook the potential to optimize sparsity allocation or rely on heuristic rules or local metrics to guide it. In this work, we observe that sparsifying attention in different layers and at different denoising steps affects final-video fidelity to substantially different degrees. Yet the identities of the most sensitive layer-step pairs remain nearly unchanged across inputs. Building on these findings, we propose Allocating Layer-step Latency by Output Tests (ALLOT), a training-free sparsity allocation method. Treating each layer-step pair as a cell, ALLOT estimates each cell's contribution to final-video error through (i) cell probing and (ii) trajectory calibration, then jointly optimizes cell densities under a measured-latency budget. It preserves existing selection strategies and kernels, replacing only the allocation component. At equal or shorter wall-clock time, ALLOT outperforms every compared allocation strategy with XAttention, with an 11.8 dB PSNR gain on HunyuanVideo. Extensive experiments across four additional published pipelines on Wan2.1, Wan2.2, and HunyuanVideo show that ALLOT outperforms or ties the default in all nine settings.