Programmable model keeps track of game worlds for longer play
Programmable World Model
Computer Vision and Pattern Recognition
Summary
Video world models can create cool visuals but often forget details about what’s going on in the game world, especially over long times. The authors designed a system where instructions are turned into computer programs that keep track of every object and their rules in a game world. This system keeps an explicit record of everything happening, even things off screen or not visible. Then it uses this record to help generate videos that match what’s going on in the game world. Their method works much better at keeping track of objects and states during long interactions than earlier methods.
What this means in practice
- •For game developers: Create games with consistent state management and predefined rules by controlling individual entities over long gameplay sessions.
- •For interactive media creators: Produce interactive video experiences that maintain coherent world states and support programmable user interactions for story-driven content.
Authors
Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang, Ruihan Yu, Muyao Niu, Siqi Yang, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang
Abstract
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.