Adaptive patch-wise denoising speeds up video diffusion models
ForcingBooster: Adaptive Patch-wise Denoising for Fast Autoregressive Video Diffusion
Computer Vision and Pattern Recognition
Summary
Making videos with AI can take a lot of computer work because every part of the video is treated the same, even if some parts are simpler to create. The authors present a way for the AI to work faster by focusing more on the tricky parts and less on the easy parts. This saves about half the computer steps needed without making the video look worse. Their method can work with many existing AI video generators and only needs a small extra component trained alongside.
What this means in practice
- •For interactive media developers: Reduce latency and computational costs when generating videos by adapting processing effort to different video regions in real time.
- •For game engine programmers: Integrate faster autoregressive video synthesis by selectively refining complex parts, improving performance in video-based scene generation.
Authors
Zekai Gu, Peng Wang, Jiahao Lu, Yongsen Mao, Cheng Lin, Yuan Liu
Abstract
Autoregressive video diffusion models have become a leading approach to real-time interactive video generation and video world modeling. However, existing methods typically allocate computation uniformly across space when generating the next video chunk, updating all patches with the same denoising schedule and number of function evaluations. This ignores the spatial heterogeneity of video content: static, simple, or recurring regions are often easier to denoise, while other regions require further refinement or additional context. A uniform budget thus leads to redundant computation on easy patches, whereas reducing the fixed step count can leave difficult patches insufficiently refined. We introduce \methodname{}, a general framework that dynamically allocates denoising steps to individual patches: easy patches stop early to improve efficiency, while difficult patches receive more refinement steps. This design accelerates generation with negligible quality degradation. Our method is compatible with most existing few-step autoregressive diffusion models, requiring only a lightweight routing module with modest training. Experiments demonstrate that \methodname{} reduces the total number of denoising steps applied across video patches by about 50\% while maintaining visual quality comparable to the four-step baseline.