MORCA improves video diffusion speed and quality with smarter caching
MORCA: Offline-to-Online Reinforcement Learning for Adaptive Cache Reuse in Video Diffusion Acceleration
Computer Vision and Pattern Recognition
Summary
Making videos with diffusion transformers can be slow because they need many steps to create each frame. To speed this up, some methods save and reuse parts of the video generation process, but deciding when to reuse or redo these parts is tricky. The authors introduce MORCA, a system that learns when to reuse or recalculate to keep video quality high while speeding up generation according to user needs. MORCA uses a smarter way to predict quality loss and adjusts caching dynamically, outperforming previous methods.
What this means in practice
- •For video generation teams: Speed up video synthesis workflows by dynamically controlling cache reuse to meet specific speed and quality requirements during diffusion model inference.
- •For graphics software developers: Integrate adaptive caching strategies in software to reduce computation time when rendering video content generated by diffusion transformers.
Authors
Yuxiang Xiong, Ruiyan Wang, Wenqiang Wang, Teng Hu, Songhang Shen, Bohao Feng, Hongqian Deng, Ran Yi
Abstract
Diffusion Transformers (DiTs) achieve remarkable performance in video synthesis, but their iterative denoising process suffers from high inference latency. To address this, caching has emerged as an effective acceleration strategy by capitalizing on inter-step redundancy during denoising. Existing dynamic caching methods typically estimate the error that cache reuse would introduce at each denoising step (step error) to guide cache decisions, whereas our concern is how much quality loss cache reuse would cause in the final generated video (terminal error). We show that step error does not directly correspond to terminal error and that latent information helps capture their relationship, thereby informing cache decisions. Moreover, existing threshold-based methods cannot provide precise speedup control, making it difficult to meet practical requirements for user-specified acceleration targets. To address these limitations, we introduce MORCA, a cache scheduling framework trained through offline-to-online reinforcement learning to make latent-aware reuse/recompute decisions under user-specified acceleration targets. Extensive experiments on different video generation models across multiple target acceleration ratios demonstrate that MORCA achieves better generation fidelity than state-of-the-art caching methods under comparable computational budgets. Code is available at https://github.com/x10ngyx/MORCA.