Video models contain correct motion data even when output is wrong
A Chosen Future Can Still Be Rewritten: Causal Writability in Video Models
Machine LearningComputer Vision and Pattern Recognition
Summary
Sometimes video models produce motions that look wrong, and it's unclear if they never learned the right motion or just aren't using it properly. The authors show that the models actually do have the correct motion inside them and that simple edits can make the videos move correctly. They discovered a sharp point in the model where changes stop affecting the output, marking when the model commits to a motion. This finding helps explain which errors can be fixed by further training and applies broadly to large video models.
What this means in practice
- •For video software developers: Fix physically incorrect video motions by applying targeted edits inside existing video models without retraining from scratch.
- •For machine learning engineers: Identify and correct motion prediction errors at specific network depths to improve training efficiency of video prediction models.
Authors
Xingyun Wang, Haomin Zheng, Man Yuan, Leqian Yang, Ziming Liu
Abstract
When a video model generates physically incorrect motion, did it fail to learn the correct motion, or did it learn it but fail to use it? We show the latter: the correct motion remains available inside the model and can still be made to control the generated video. We train on videos where red masses oscillate slowly and blue masses oscillate quickly, then test a red mass with fast observed motion. Even when the model generates slow motion in this conflicting case, a low-dimensional edit predicted from simple physical variables restores the correct fast motion. We call this ability causal writability. At fixed strength, we find a sharp depth boundary: the same edit changes the video before the boundary but not after it. This closure marks commitment for that write. The motion signal nevertheless remains, and a stronger downstream write can restore physical motion, while excessive gain overshoots. Early causal writability predicts which errors training later corrects: those errors are writable at more network depths than errors that persist. We reproduce both causal writability and its sharp closure in a pretrained 1.3B video model, supporting generality across model scale and training regime.