Causal video diffusion improves by limiting reliance on past frames
Conditional Residual Prediction: Improving Autoregressive Video Diffusion without a Bidirectional Teacher
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Video models that create frames one after another often perform worse than models that look at the whole video at once. The authors found a new way to train these step-by-step models so they don’t depend too much on earlier frames when making new ones. Their method makes the model first try to predict the next frame without using past frames, then adds any extra needed details from the past. This approach improves video quality and scales well without needing extra big models to guide training.
What this means in practice
- •For video streaming engineers: Build autoregressive video generators that produce high-quality streaming video without requiring complex teacher models.
- •For mobile video app developers: Create efficient on-device video generation tools that progressively produce frames with improved quality and scalability.
Authors
Bowen Zheng, Zhiguang Liu, Jiarong Ou, Rui Chen, Tianyang Hu
Abstract
Causal video diffusion models generate video autoregressively, which suits streaming, interactive, and long-video generation. Under standard training, however, they often yield lower generation quality than bidirectional models of the same size. Many existing approaches address this gap by initializing from or distilling a pretrained bidirectional teacher. We instead train a causal model from an image-model initialization, with no bidirectional video model at any stage. Because this path requires neither a large bidirectional teacher nor a complex distillation pipeline, it is simpler and more scalable. On this path, we find that a causal model trained on ground-truth history becomes strongly dependent on it, so that at inference errors in its own generated history propagate forward. We hypothesize that much of this dependence is unnecessary, because the current input already determines much of what the history provides. We propose Conditional Residual Prediction (CRP), a simple recipe for reducing a model's reliance on a condition: the model first predicts the target without the condition, and the condition may only add a residual on top of this prediction. Applied to history, CRP makes the model predict each chunk from the present as far as it can and use the past only for what the present cannot supply. In controlled experiments, CRP nearly closes the 6.14-point gap to a bidirectional model trained under the same setup. Scaling this recipe, we train Optica, a 2B-parameter causal video model that autoregressively generates 5-second 480p videos and reaches 82.78 on VBench with only about 15M training videos.