Video-Mirai: Autoregressive Video Diffusion Models Need Foresight
2026-06-02 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors point out that video generators that predict frames step-by-step often only learn to explain the current frame, missing important info for making future frames consistent. They propose Video-Mirai, a new training method where a special encoder looks at the whole generated video to teach the model better future-aware features, but this only happens during training and doesn't slow down actual video generation. Their method improves video quality and consistency on tested benchmarks, showing that teaching the model about the future helps it make better videos. They emphasize that looking ahead should guide how the model learns representations, not how it generates frames.
autoregressive video generationvideo diffusionrepresentation-level planning gapcausal inferencevideo rolloutfrozen foresight encoderdistillationKV-cachefuture-conditioned targetsvisual autoregressive models
Authors
Yonghao Yu, Lang Huang, Runyi Li, Zerun Wang, Toshihiko Yamasaki
Abstract
Causal video generators must predict from the past, but they need not learn only from it. In streaming autoregressive video diffusion, each emitted segment becomes a commitment that future segments must preserve. Standard training, however, only asks each causal state to explain the present. This creates what we call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future. We introduce Video-Mirai, a training-only method that closes this gap without changing causal inference: the generator rolls out causally, a frozen foresight encoder reads the completed rollout non-causally, and a lightweight predictor distills the resulting stopped-gradient targets into causal states. Future frames supervise representations, never generator inputs. At inference, the encoder and predictor are discarded, leaving the original architecture, per-step FLOPs, and KV-cache behavior unchanged. Video-Mirai improves a strong Causal-Forcing baseline on 5-second VBench from 83.8 to 84.6 in terms of Total Score. On 30-second rollouts beyond the training horizon, subject consistency improves from 84.9 to 88.5 and background consistency from 90.2 to 91.9. Ablations identify future-conditioned targets as the key ingredient, and probes show that future frames become more decodable from current features. Causality should constrain inference, not representation supervision. Our study highlights that visual autoregressive models need foresight. Project page: https://y0uroy.github.io/Video-Mirai.