Video generation uses adaptive attention to save computation
Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion
Computer Vision and Pattern Recognition
Summary
Generating videos with computers is very slow because the method of paying attention to every part of the video is expensive. The authors propose a new way called HetA-DiT that spends more computing effort only on the tricky parts while handling simpler parts faster. A small extra system guesses which parts are hard to create and guides this process, allowing the method to keep quality high but use much less computing power. They tested this on different video generation models and datasets and found it keeps video quality good while using much less heavy computation.
What this means in practice
- •For video generation developers: Integrate adaptive attention strategies that focus computing power on complex video areas to reduce resource use while maintaining output quality.
- •For graphics hardware engineers: Optimize hardware or software pipelines to support dynamic routing of tokens for efficient video diffusion models using heterogeneous attention.
Authors
Olga Zatsarynna, Denis Korzhenkov, Juergen Gall, Amir Habibian, Mohsen Ghafoorian
Abstract
Efficient video generation requires reducing the quadratic cost of self-attention over long spatio-temporal token sequences. Existing efficient-attention methods typically apply the same computation pattern to every token, even though denoising difficulty varies substantially across video regions and evolves throughout the generation process. We introduce HetA-DiT, a heterogeneous attention mechanism that adaptively allocates computation according to token difficulty. A lightweight uncertainty branch predicts a token-wise estimate of denoising difficulty, which is used to route uncertain tokens through dense global attention while processing more reliable tokens with efficient local attention. The resulting routing is content- and timestep-adaptive, retains global context where it matters most, and provides a single parameter for controlling the quality-efficiency trade-off. HetA-DiT is compatible with few-step distribution-matching distillation and introduces no additional Transformer evaluation at inference time by reusing uncertainty estimates from the preceding denoising step. We evaluate the method on DMD-distilled Wan2.2-5B and Wan2.1-1.3B models. Across VBench, VBench-2.0, and human preference evaluation, HetA-DiT maintains competitive generation quality while routing only approximately 20% of tokens through dense attention.