Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models
2026-08-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors investigate how video prediction models can imagine what would happen if different actions were taken partway through a generated video, not just predict what actually happens. They introduce a method where two video sequences share the same starting situation and random events but differ only at a specific action point, allowing exact comparison between what actually happened and what could have happened. This helps measure how changes spread through time and space in the video, using exact simulator data for verification. Their work provides a new way to test and improve models on generating realistic alternate scenarios, with experiments to come.
interactive video world modelsautogressive rolloutcounterfactual generationnoise-coupled rolloutsexogenous noisePearl's counterfactual procedureminimal-change principlespatiotemporal localitysimulator state forkingpost-training rewards
Authors
Yu Ma, Hongli Shi, Xinran Xu
Abstract
Interactive video world models generate rollouts autoregressively under an action stream, yet they are trained and evaluated almost exclusively on factual prediction. We study counterfactual generation inside the rollout: given a trajectory the model has itself generated, what would have happened had the actions differed from step t* onward? We formalize noise-coupled twin rollouts --- a factual and a counterfactual branch sharing the generated prefix and the future exogenous noise sequence, diverging only in the action stream at an intervention point. Because the factual branch is self-generated, its exogenous noise is known exactly: the abduction step of Pearl's counterfactual procedure is exact by construction, sidestepping the approximate-inversion problem faced by editing-based pipelines. Noise coupling further turns the minimal-change principle into a per-sample verifiable property: we define a spatiotemporal locality metric that penalizes divergence outside the causal descendants of the intervention, computable against simulator ground truth without a learned judge. Forking the simulator state at t* yields ground-truth counterfactual re-renders, which we use as verifiable rewards for post-training. This note establishes the formal framework, metric definitions, and positioning; experiments are forthcoming.