Video diffusion acceleration improves with feedback driven caching
Heuristics are Not Enough: Calibrated and Corrected Caching for Video Diffusion Acceleration
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating videos using diffusion models is often slow because the computer has to process many steps. Previous methods sped this up by storing and reusing some parts during generation, but they used simple rules that didn't always work well and caused errors to add up. The authors created AlignCache, a smarter system that watches how caching decisions affect video quality and adjusts its approach automatically for different phases of the process. This feedback-based method helps generate videos faster without losing quality.
What this means in practice
- •For video generation engineers: Speed up video output in diffusion-based generation pipelines while maintaining visual quality through adaptive caching.
- •For graphics hardware designers: Incorporate feedback-driven cache management to optimize resource usage during complex video synthesis tasks.
Authors
Jiyue Jiang, Zhiyu Huo, Yuhan Chen, Yanruisheng Shao, Gong Zhang, Lei Deng
Abstract
Feature caching has become a mainstream training-free strategy for accelerating video diffusion models by reusing intermediate outputs across denoising steps. However, existing adaptive methods rely on heuristic difference measures misaligned with actual generation quality, employ a single global threshold that ignores heterogeneous phase dynamics, and never validate their budget estimates against true approximation error, allowing systematic bias to propagate uncorrected. We propose \textbf{AlignCache}, a closed-loop caching framework that directly aligns scheduling decisions with their true influence on the final generated video through three tightly coupled components: (1) phase-adaptive budget routing that partitions the denoising trajectory into structurally distinct phases, each governed by an independent threshold and a staleness-aware guard; (2) closed-loop threshold calibration that exploits counterfactual error measurement at the classifier-free guided output level to adjust decision boundaries online via an EMA feedback loop; and (3) momentum-augmented draft synthesis combining direction-magnitude gated residual extrapolation with anchor-guided unconditional correction. By transforming scheduling from a static risk accumulator into a feedback-driven, self-correcting controller, AlignCache consistently achieves superior acceleration-quality trade-offs across state-of-the-art video generation models. Our code is available at https://github.com/jiangjyjy/AlignCache.