Veda: Scalable Video Diffusion via Distilled Sparse Attention
2026-05-28 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of slow video generation in diffusion transformers caused by costly self-attention steps. They find that the key to maintaining quality is how well the simplified attention focuses on important parts, rather than just how much it cuts down on connections. They propose Veda, a system that smartly chooses attention areas based on reconstructing full attention patterns and uses efficient hardware tricks to speed up processing. Tests show Veda makes video generation much faster without losing quality, especially as video length and resolution grow.
diffusion transformersself-attentionsparse attentiontile-wise geometryvideo generationreconstruction problemhardware accelerationWaver-T2Vspatiotemporal resolutionsequence length
Authors
Shihao Han, Hao Yang, Xinting Hu, Xiaofeng Mei, Yi Jiang, Xiaojuan Qi
Abstract
Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1$\times$ end-to-end speedup and a 10.5$\times$ self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.