Selective competition domains improve video diffusion transformer efficiency
Competition Domains Matter: Selective Head Globalization for Training-Free Sparse Attention in Video Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
Video diffusion transformers help make realistic videos but are slow because they look at many parts of a video at once. The authors found that letting different parts of the model choose their focus area differently makes the process faster and produces better video quality without needing additional training. They created a method called EG-Sparse that picks smaller or larger groups to pay attention to based on how diverse the data is at each head, improving speed and quality. This shows that adjusting which parts compete during attention is important for efficient video processing.
What this means in practice
- •For video processing teams: Speed up and improve quality of video generation pipelines by using adaptive selective attention domains per attention head.
- •For machine learning engineers: Reduce GPU time and memory use during inference in transformer-based video models with selective sparse attention routing.
Authors
Tiantian Liu, Meng Wan, Zhiqiang Liang, Jue Wang, Ningming Nie, Yangang Wang
Abstract
Training-free sparse attention is a practical route to efficient video diffusion transformer inference, yet existing methods typically fix the competition domain at each query row for every attention head. Prior work primarily addresses how to estimate token-level importance within this fixed domain; whether the domain itself should adapt to head heterogeneity remains under-specified. We introduce EG-Sparse, a training-free framework that routes low-entropy heads to a grouped cross-head pool while retaining per-row local selection for high-entropy heads, using only scores already materialized by the sparse pipeline. A controlled diagnostic fixes the ranking score and varies only the competition rule: grouped cross-head selection reduces low-entropy-head reconstruction error by 9.8%, whereas forcing high-entropy heads into an all-head global pool increases it by 7.9%. Across four configurations, EG-Sparse achieves higher video quality while reducing attention density and GPU denoising latency compared with strong training-free sparse-attention baselines. Mechanism ablations further show that selective domain adaptation outperforms uniform, random, and fixed domain-assignment policies. These results support competition-domain adaptation as a distinct design axis within centroid-guided training-free sparse attention.