AI summaryⓘ
The authors study video world models that simulate scenes based on actions, focusing on whether these models really keep track of what’s happening behind the scenes. They use a test called the video Shell Game, where objects swap places, and find that many common models can mimic short sequences but fail to keep track of longer swaps accurately. They show that models need an internal way to remember and update the hidden state, not just rely on generating realistic-looking images. Some advanced methods that update this internal state more flexibly perform better at tracking long sequences. The authors also explore more complex tasks and discuss how this understanding can help build better video models that really understand and remember their environments.
video world modelhidden stateTransformerlinear attentionShell Gamestate trackingautoregressive modelsfast weightseigenvaluesdiffusion model
Authors
Joonghyuk Shin, Yicong Hong, Jaesik Park, Xun Huang
Abstract
Video world models are increasingly used as simulators, yet visual fidelity alone does not show that a model maintains the hidden state of the world. We examine this gap with an action-conditioned video Shell Game, a visual analog of $S_5$ state tracking that decouples visual rendering from compositing the hidden state underneath. Bidirectional and autoregressive Transformers, Mamba, and linear attention restricted to nonnegative transition eigenvalues all fit the training horizon of 5 swaps and then fall toward chance on longer swap chains (extrapolation) while still rendering plausible video with additional denoising steps providing no benefit. The pixel-based diffusion target never supervises the unseen hidden state, so the generated frames cannot carry it and the state has to live inside the architecture rather than in the tokens. For a Transformer, that architectural state is only an append-only KV cache, so the model has to re-derive the hidden arrangement from the whole history at every chunk. We find two mechanisms that do extrapolate, and both carry a state across chunks and revise it in place. Linear attention succeeds once its transition eigenvalues may be negative, and TTT with a nonlinear fast weight succeeds by updating the feature map through which it reads its own state. We further examine harder cases in dynamic world exploration tasks, and discuss the broader implications for building stateful video world models.