Autoregressive video models improve long videos by separating motion from frames
Disen-Forcing: Disentangling Semantic Anchoring from Motion for Autoregressive Video Diffusion
Computer Vision and Pattern RecognitionMachine Learning
Summary
Generating long videos in real time is hard because small errors add up and make the video worse over time. The authors found that in existing models, only some parts of the system really use the first few video frames to keep the video consistent, while others focus on short-term motion. They created a new way to teach these models that keeps the parts specialized and better at capturing motion, making longer videos look more consistent and lively.
What this means in practice
- •For video generation engineers: Improve the quality and consistency of long generated videos by using disentangled context routing during model training.
- •For visual effects studios: Produce longer high-quality synthetic video sequences with more realistic motion using refined autoregressive diffusion models.$Commercial implications: Enables new video generation services with improved long-range video quality, appealing to studios seeking better visual effects automation.
Authors
Zhengyang Yu, Akio Hayakawa, Masato Ishii, Qingtao Yu, Xuesong Li, Takashi Shibuya, Yuki Mitsufuji, Jing Zhang
Abstract
Autoregressive video diffusion models (AR-VDMs) enable real-time, long-horizon video generation but degrade rapidly once the inference horizon outruns the training horizon, due to compounding exposure bias. To stabilize long rollouts, recent methods heuristically adopt frame sink, keeping initial frames as a persistent model context, inspired by attention sinks in LLMs. We revisit this design and find that the mechanism by which initial frames stabilize long-range consistency in AR-VDMs is not an LLM-style attention sink; instead, long-range consistency is anchored through a sparse subset of attention heads, while the majority of heads predominantly capture local temporal dynamics. This observation reveals an over-conditioning pathology in existing frame-sink strategies: by universally exposing every head to the pinned initial-frame context, the model becomes prone to collapsing motion toward the low-dynamic modes of the teacher during model distillation. We propose Disen-Forcing, a distillation-time plug-in for AR-VDMs that disentangles and reinforces this head specialization through disentangled context routing and motion-disentangling regularization. On long-horizon video generation benchmarks, Disen-Forcing simultaneously improves long-range consistency and motion dynamics of base AR-VDMs, a trade-off that sink-based methods fail to balance.