BoxWorld separates interaction rules from video rendering for real-time control
BoxWorld: Let World Engines Control and Video Generators Render
Computer Vision and Pattern Recognition
Summary
Creating interactive video worlds where objects behave as users want is hard because existing systems mix up how objects act and how they look. The authors propose BoxWorld, which splits the job: one engine follows object rules and updates states, while a separate renderer makes the images you see. This makes it easier to control object actions and see their effects in real time, like in games. They teach the system using videos of gameplay and other footage to learn how objects should move and change.
What this means in practice
- •For game developers: Create real-time interactive game worlds where object behaviors follow clear rules separate from rendering graphics.
- •For video content creators: Produce videos that respond dynamically to user control by manipulating object states separately from visual effects.
Authors
Zike Wu, Ahmed Hatem, Chintan Trivedi, Wallace Michel Pinto Lira, Xuebin Qin, Lele Wang, Renjie Liao
Abstract
We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.