Decoupled sparse attention speeds up video diffusion without training

DecoupAttn: Decoupled Sparse Attention for Training-Free Acceleration of Video Diffusion

Computer Vision and Pattern Recognition

Summary

Processing videos with diffusion models is slow because it requires computing many attention steps. The authors found a way to skip less important parts and correct the errors efficiently without retraining the model. They do this by splitting attention work between fixed important parts and groups that compensate for skipped parts over time. This approach is set up once using calibration videos and then speeds up video generation during use.

What this means in practice

  • For video processing engineers: Accelerate video generation models by reducing attention computations without retraining, improving inference speed on existing diffusion transformer implementations.
  • For machine learning platform teams: Implement calibrations for decoupled sparse attention to cut resource use during video model deployment while maintaining output quality.

Authors

Yanjin Li, Bingbing Qiu, Qiang Huang, Rui Qian, Yanfeng Li, Jian Wang, Xiaoying Tang, Mingda Zhang

Abstract

Sparse attention accelerates few-step video diffusion transformers by computing a few key blocks exactly and compensating for the rest. However, existing sparse attention still faces key challenges, including identification overhead from finding critical blocks at every denoising step, suboptimal budget allocation that sets sparsity budgets uniformly or by local metrics, and cross-step error accumulation as reused masks or clusters go stale. Observing that the errors left on skipped blocks cancel each other once every key is compensated, we propose DecoupAttn, a new paradigm of decoupled sparse attention: semantic clusters compensate every key, and exact attention corrects their error on a few blocks fixed once offline by position. To support this paradigm, we introduce Distortion Budget, a rate-distortion objective for budget allocation that places exact computation across timesteps, layers, and heads where it removes the most distortion that compensation leaves in the final latent. Built on this budget, we further propose Reassign Cache, in which each step reassigns the drifting keys to cached centroids and refreshes compensation while the budgeted blocks stay fixed. Moreover, DecoupAttn is training-free: a few calibration clips compile one plan offline, and inference runs standard block-sparse kernels with one fused compensation kernel. Our code will be released after submission.