Rollout-marginal distillation improves long video predictions quality
Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Making videos one piece at a time is fast but can cause mistakes to build up over time, ruining the video quality. The authors introduce a way to check and fix these pieces separately, so each part looks better on its own. They also add a second step to make sure the whole video still flows smoothly. This method helps keep video quality high, even for long videos, better than older methods.
What this means in practice
- •For video game developers: Generate longer, high-quality streamed game cutscenes without increasing latency from error build-up in video prediction.
- •For media streaming platforms: Improve video generation that adapts live or low-latency content streams by reducing degradation over long videos.
Authors
Chenjian Gao, Zhihao Hu, Jianqi Ma, Jun Zhang, Weidong Zhang, Tianfan Xue
Abstract
Autoregressive (AR) video diffusion enables low-latency, streamable video generation, but prediction errors often accumulate over long rollouts. Training the generator on its own rollouts exposes it to these imperfect histories. However, existing video-level distribution matching distillation (DMD) scores the whole rollout jointly. Because a chunk is evaluated together with its past and future, its correction can favor matching artifacts in the surrounding context merely to preserve temporal consistency. To provide a clearer visual-quality signal, we introduce Rollout-Marginal Distillation (RMD). RMD retains the generated history for AR prediction but scores each chunk independently against a chunk teacher, ensuring its quality correction is not compromised by an imperfect temporal context. To compensate for the lack of temporal context in independent chunk scoring, RMD subsequently applies video-level DMD to restore temporal coherence. Extensive experiments demonstrate that RMD maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines. Code and video results are available at https://cjeen.github.io/RMD