Mask forcing improves real time video generation with fewer artifacts
Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
Computer Vision and Pattern Recognition
Summary
Generating videos by predicting frames one after another can sometimes produce videos that look too smooth or overly bright, reducing their realism. The authors found that a common way to teach these video generators can cause them to focus too narrowly on certain typical video frames, missing the variety of real videos. They introduced a new method called Mask Forcing that mixes in some cleaner parts while training, encouraging the model to consider more possibilities and improve its predictions. This approach leads to higher quality videos without needing extra training steps or real video data.
What this means in practice
- •For video game developers: Enhance real-time video generation quality in games by reducing oversaturation and smoothing artifacts in AI-generated scenes.
- •For mobile app developers: Improve on-device video synthesis apps with better frame prediction that maintains visual realism without requiring large datasets or extra training.
Authors
Zhuoran Zhao, Shengju Qian, Tongtong Liang, Xianghao Kong, Songchun Zhang, Junchao Huang, Guian Fang, Xin Wang, Pan Hui, Anyi Rao
Abstract
Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.