Efficient attention speeds up text to image and video generation
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
Generating images and videos from text using diffusion transformers is powerful but slow because of a time-consuming step called dense attention. The authors found that some parts of this attention don't change much and can be reused to speed things up without retraining the model. They created a new method called Sink-Sparse Hybrid Attention that mixes fixed attention patterns with a smaller, quicker component to keep quality high but run faster. Their approach almost halves the runtime in tests while making images and videos that look just as good.
What this means in practice
- •For ai model engineers: Speed up inference of text-to-image and text-to-video diffusion transformer models without retraining or quality loss by reusing attention patterns identified during calibration.
- •For game developers: Generate high-quality images and videos from text prompts faster in creative game content pipelines by integrating efficient attention methods.
Authors
Bingxin Xing, Yu Zhang, Jiatong Ding, Lei Lu, Ping Wang
Abstract
Diffusion Transformers (DiTs) have become a common backbone for high-quality text-to-image and text-to-video generation, but repeated dense self-attention remains a major inference bottleneck as token sequences grow and the same operator is invoked across many denoising steps. Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations. This leads to Sink-Sparse Hybrid Attention, a train-free attention replacement that reuses static sink masks and supplements them with a lightweight pooled-KV branch, while leaving model weights and sampling schedules unchanged. On Wan2.1-T2V-1.3B, the method attains 1.54$\times$ end-to-end speedup while maintaining generation quality. On Z-Image-Turbo, it attains 1.57$\times$ end-to-end speedup with competitive GenEval2 performance. These results indicate that reusable importance-aware sparse support can provide an effective train-free acceleration path for both T2V and T2I DiTs.