Frequency step cache speeds up autoregressive video diffusion with better quality
Frequency-Step Cache: Separating Synchronization and Repair for Efficient Autoregressive Video Diffusion
Computer Vision and Pattern RecognitionMachine Learning
Summary
Autoregressive video generation involves predicting video frames one after another, which can be very slow and computationally expensive. The authors present Frequency-Step Cache (FSC), a new method that cleverly separates when to fully recompute frames and how to fix small errors in between. This separation allows the system to run faster while maintaining or improving video quality compared to previous caching methods. They tested FSC on multiple video datasets and found it improves speed and quality trade-offs consistently.
What this means in practice
- •For video software developers: Implement frequent full updates with selective repairs in video diffusion models to speed up long-horizon video generation while improving output quality.
- •For ai inference engineers: Optimize large-scale autoregressive video model deployment by using frequency-based cache control to reduce computing costs without sacrificing fidelity.
Authors
Kerry Xu, Param Misra, Zachary Stenglein, Athri Karunamurthy, Fanjiang Ye, Yuke Wang
Abstract
Autoregressive (AR) video generation enables long-horizon synthesis but requires repeated forward passes through large diffusion transformers. Training-free caching reduces this cost by reusing intermediate computation, yet existing controllers often couple two decisions: when to synchronize the cache exactly and how much computation to repair between synchronizations. We introduce Frequency-Step Cache (FSC), which separates these roles. FSC schedules exact full-computation anchors on a dense-to-sparse trajectory, while a lightweight latent-space risk based on frequency-resolved drift and token age controls inter-anchor repair through whole-step replay, selective token recomputation, or full repair. On SkyReels-V2, whole-step reuse provides most of the acceleration, while selective token computation is most effective as fidelity repair. A controlled ablation shows that using one reactive signal for both synchronization and repair reduces fidelity at matched latency, with the same separation advantage observed on FramePack and Matrix-Game. Across multiple autoregressive video backbones, FSC improves speed-quality trade-offs. On SkyReels-V2, FSC-Speed reaches 2.36x speedup versus MotionCache's 2.275x while improving SSIM by 0.019 and PSNR by 0.5 dB. At MotionCache-matched speed, FSC-Quality improves SSIM by 0.031 and PSNR by 1.0 dB and lowers LPIPS by 0.024. Without retuning, FSC-Quality transfers to SkyReels-V2 14B at 2.12x speedup versus 1.79x for MotionCache.