Summary
Sometimes, generating videos that react correctly to user actions is hard when the video can’t show everything important about the scene’s state. The authors created a system called Stateful Harness that keeps track of hidden information separately from the video, making sure actions follow rules and create the right outcomes. Their system uses a world state that updates according to rules and then makes video clips based on that state, improving how realistic and rule-following the videos are. They also made a benchmark to test how well different methods follow rules when hidden state matters. Their approach greatly increased the accuracy of producing rule-compliant video outcomes compared to existing methods.
What this means in practice
- •For game developers: Create interactive game scenes that follow persistent rules even when some effects depend on hidden state.$Commercial implications: Enables more realistic and consistent game environments by separating rule execution from visual rendering, useful for game engines and interactive storytelling.
- •For simulation software engineers: Build simulations that accurately model rule-governed events by maintaining clear world state separate from video output.
Authors
Chan Yan Ki, Yanran Zhang, Wenzhao Zheng, Xiaofeng Wang, Zheng Zhu, Lei Chen, Jie Zhou, Jiwen Lu
Abstract
Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.