Video generation speed improved with reciprocal sparse attention and caching
RECAP: Reciprocal Sparse Attention and Hierarchical Caching for Efficient Video Diffusion
Computer Vision and Pattern RecognitionMachine Learning
Summary
Generating videos with diffusion models can be very slow because of the computation needed. The authors noticed that using sparse attention combined with caching data across steps speeds things up, but errors from sparse attention can build up over time. They developed a method called RECAP that fixes these errors by alternating between quick, rough guesses and occasional detailed corrections without extra training. This approach makes video generation much faster while keeping the final video quality high.
What this means in practice
- •For video generation developers: Speed up video synthesis by integrating RECAP’s correction and caching to improve quality without increasing computation.
- •For machine learning infrastructure teams: Reduce computational costs for video diffusion inference by applying hierarchical caching and error feedback as in RECAP.
Authors
Tiange Xu, Kai Liu, Linghe Kong, Yulun Zhang
Abstract
Diffusion transformers generate high-quality videos but remain computationally expensive. Sparse attention and temporal caching reduce computation within and across denoising steps, respectively, offering a natural composition. However, predictions produced with sparse attention become sources for temporal reuse, coupling their errors along the sampling trajectory. We observe that improving the accuracy of these sources recovers fidelity without changing the reuse schedule, and that sparsification errors remain directionally aligned across nearby steps, suggesting that correction information can be shared rather than recomputed. These observations motivate RECAP, a training-free method that couples sparse attention with hierarchical caching through reciprocal correction, adaptively coordinated by online error feedback. Cached dense refinement corrects omitted interactions in current sparse estimates, while fresh sparse statistics track input changes and regulate the aging correction. The resulting predictions then renew the sources for whole-step prediction reuse, allowing prediction history to advance between dense refreshes. We evaluate RECAP on multiple video diffusion models against sparse-attention methods, caching methods, and their direct compositions. On HunyuanVideo at 720p 5-second generation, RECAP achieves a 3.05x speedup and outperforms all tested direct compositions in both fidelity and inference speed.