MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation
2026-06-08 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors address the problem of making video-generating AI keep things consistent over many frames, which is hard because processing long sequences is costly. They train a system that first breaks each video frame into different levels of detail, from rough outlines to fine textures. Then, their video model generates videos step-by-step, starting with coarse details and adding finer ones later, focusing computing power where it matters most. Tested on long Minecraft videos, their method maintained object and scene consistency much better than previous approaches.
video generative modelslong-range consistencytransformer modelsautoencodermulti-scale tokensdiffusion modelcoarse-to-fine generationlatent spaceobject permanenceMinecraft dataset
Authors
Ishaan Preetam Chandratreya, David Charatan, Basile Van Hoorick, Sergey Zakharov, Vitor Guizilini, Phillip Isola, Vincent Sitzmann
Abstract
Video generative models have become increasingly powerful, but long-range consistency remains challenging to achieve because even a few dozen frames require impractically long transformer sequence lengths. We show that this issue can be mitigated by generating video using coarse-to-fine rollout within a multi-scale token space. Our approach is simple: first, we pre-train an autoencoder that compresses each frame into a hierarchy of tokens, with levels ranging from the typical latent resolution to only a handful of tokens per frame. The coarsest levels capture the most consequential information, such as scene layout and semantics, while finer levels add high-frequency appearance and texture. Then, we train a video diffusion model to generate these tokens using coarse-to-fine rollout. By carefully controlling the level of detail at which frames are generated and used as context during each rollout step, we are able to preserve long-range consistency in geometry and object permanence while spending less compute on the long-range consistency of less perceptually relevant details. We validate this approach using a custom dataset of long Minecraft videos, where it produces substantially more consistent rollouts compared to existing baselines.