Video generation speeds up with mixed precision and draft attention
DraftAttention2: Fast Video Diffusion with Low-Resolution-Guided Mixed-Precision Attention
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating videos with AI usually takes a lot of computing power, especially because the system needs to pay attention to many small parts spread across space and time. The authors created DraftAttention2, a method that first looks at a low-resolution version of the video to figure out which parts are most important to focus on. It then assigns more computing detail (precision) to key parts and less to others, skipping some areas to save time without much loss in quality. Their approach blends fast approximate attention with precise calculations where needed, leading to better quality videos generated faster.
What this means in practice
- •For video content creators: Generate higher quality videos faster using efficient attention models that balance detail and speed across frames.
- •For graphic software developers: Integrate mixed-precision attention techniques to accelerate video generation features in creative applications without retraining models.$Commercial implications: Enables faster and higher-quality video effects and synthesis features in commercial creative software products.
Authors
Rui Ding, Haopeng Li, Weize Ma, Yufa Zhou, Yitong Li, Xiaoling Zhou, Jiashuo Cao, Liyang Li, Hua Geng, Jiuxiang Gu, Jun Lin, Enze Xie, Xuan Shen
Abstract
Video generation has broad applications in content creation and entertainment. Diffusion transformers have advanced the quality of generated videos, but attention over spatiotemporal tokens becomes increasingly expensive as video resolution and duration increase. We present DraftAttention2, a training-free framework that uses the low-resolution draft attention map to jointly select attention blocks and assign their numerical precision. Specifically, spatial 2D average- and max-pooled queries and keys capture complementary regional statistics to estimate block importance, and a shared ranking assigns higher precision to important blocks, lower precision to less important retained blocks, and skips the rest under configurable budgets. Our analysis separates sparsification error from attention-weighted quantization error, establishing when recovering skipped interactions with low-bit computation tightens the output-error bound. This analysis motivates retaining more interactions at low precision while reserving higher precision for blocks with larger attention mass. To translate these fine-grained assignments into practical speedups, we further develop fused operand preparation and a single attention kernel with precision-specific phases, sharing data movement, softmax statistics, and output accumulation across precisions. Experiments demonstrate that our method achieves a superior quality-efficiency trade-off over existing efficient video generation methods. Notably, its advantage is particularly pronounced for few-step video diffusion, where jointly combining sparsity with 4- and 8-bit mixed-precision computation substantially improves generation quality while retaining significant acceleration. Code is available at https://github.com/anemoi-project/anemoi