New method improves consistency in long video generation
Reference Forcing: Long Video Generation with Region-Guided Drift Correction
Computer Vision and Pattern Recognition
Summary
Generating long videos is hard because over time, characters and backgrounds can change in unwanted ways, called drift. The authors designed a technique called Reference Forcing to keep these elements consistent by always comparing new video parts to initial reference frames. They use a system that learns how to keep focus on specific regions and corrects past mistakes during generation. Tests show their method better maintains consistent scenes and characters even in very long videos.
What this means in practice
- •For video game developers: Generate long, smooth in-game cinematics without visual inconsistencies or background drift over time.
- •For advertising production teams: Create extended promotional videos that maintain consistent branding elements and backgrounds throughout.
Authors
Sheng Zhao, Haitian Zheng, Weikai Lin, Junjie Zhao, Yuhao Zhu
Abstract
Long video generation should keep the subject and background consistent while supporting rich, natural motion. However, autoregressive models are often distilled with distribution matching distillation (DMD) over short windows, which does not train them to keep this consistency over long ranges. As a result, the subject and background can drift over time. Methods that manage the key-value (KV) cache retain or retrieve history to reduce drift. Yet this history may itself have drifted. We propose Reference Forcing, which uses the initial frames as an appearance reference. We first build a teacher that uses attention masks to force each region of a new frame to attend to the same region in the initial frames. The teacher finds these regions with off-the-shelf segmentation models. We then distill this behavior into a separate attention path of the student, which needs no external models at inference. We further introduce value correction, which corrects drifted values in the KV cache toward the matching values of drift-free initial frames. Experiments show that Reference Forcing achieves state-of-the-art subject and background consistency among long-video methods on different base models, and its advantage grows with video length. Ablations show both components are needed to gain consistency without losing motion. More visualizations and code are available at https://zhaosheng-thu.github.io/reference-force/