Video generation speeds up by focusing on moving parts only
Dynamics-Aware Sparse Attention for Efficient Autoregressive Video Diffusion
Computer Vision and Pattern Recognition
Summary
Generating videos frame by frame with computers can be very slow because the process treats every part of the video equally, even parts that don’t change much. The authors propose a method called Dynamics-Aware Sparse Attention that helps computers focus their work on parts of the video that are actually moving and change over time. This approach reduces the computer’s memory use and speeds up the video generation by over twice as fast while still keeping the video quality high. They tested their idea on a video dataset and showed it can work well.
What this means in practice
- •For video software engineers: Generate long videos more efficiently by reducing computation on static parts to speed up workflows without losing quality.
- •For computer graphics developers: Improve video synthesis tools by integrating motion-aware attention mechanisms that minimize resource usage during rendering.
Authors
Yuanyu He, Zhuokun Chen, Yefei He, Zhiwei Tang, Jiasheng Tang, Yinghao Yu, Jianfei Cai, Bohan Zhuang
Abstract
Autoregressive Diffusion Transformers have emerged as a powerful paradigm for long video generation. However, they are often constrained by prohibitive computational overhead and substantial memory footprints. This inefficiency arises from dense computation mechanisms that allocate uniform processing resources across all spatiotemporal regions, failing to leverage the inherent temporal redundancy of video where motion is typically confined to sparse areas. In this paper, we propose Dynamics-Aware Sparse Attention (DASA), a unified framework that shifts from dense computation to sparse, motion-driven generation to eliminate both storage and computational redundancies. Inspired by video codec principles, our approach explicitly decouples video content into dynamic and static regions. To mitigate memory bottlenecks, we selectively compress the historical KV cache by retaining only dynamic features within the generated chunk. Furthermore, to reduce computational overhead, we employ a region-level computation allocation strategy based on localized dynamics. Additionally, a dynamics-guided distillation method is introduced to enhance performance via a lightweight distillation training process. Experiments on VBench demonstrate that our approach accelerates autoregressive video generation by 2.16x, while preserving high visual fidelity.