Generative simulation links world state with immersive first-person experiences
Generative World Simulation with DAO and JING
Computer Vision and Pattern Recognition
Summary
Interactive worlds often struggle to keep track of past actions or change realistically over time. The authors present a system that keeps a shared, up-to-date model of everything in the world, including characters and objects, and uses this to create personalized audiovisual experiences for each user. People can explore, interact, and talk with characters, and their actions change the world and what they experience next. This system helps build virtual worlds that remember history and respond dynamically to interactions.
What this means in practice
- •For virtual reality developers: Create VR worlds that maintain consistent state over time and generate personalized audiovisual scenes responsive to user actions.$Commercial implications: Enables VR software companies to offer interactive, evolving virtual environments with persistent user-driven histories enhancing immersion.
- •For game designers: Build games where player actions have lasting consequences in a shared world and influence ongoing character and environment behavior.
Authors
XGEN Team, Guohao Cai, Haoxuan Che, Frank Fan, Chenran Huang, Junjia Huang, Jiayi Jiang, Yanheng Li, Leon Liu, Shaoteng Liu, Xi Liu, Feng Nie, Zhefan Rao, An Wang, Hanwen Wang, Pengxiang Yan, Jieming Yu, Qian Zha, Wentao Zheng
Abstract
We present a Generative World Simulation framework that couples persistent shared world state with continuous first-person audiovisual generation. The framework integrates DAO, a computable world engine, and JING, an Egocentric Interactive Experience Model, to support sustained interaction for humans and agents. DAO maintains the states of characters, objects, and spaces and their relationships, advancing the world according to interaction rules and world time while preserving the consequences of past actions. From this evolving state, DAO derives observer-specific conditions for JING, which combines these conditions with character, object, and scene references, observation history, and current actions to generate each observer's audiovisual experience. Users can explore environments, manipulate objects, and communicate with characters through viewpoint control and text instructions, with their actions shaping both the shared state and subsequent experiences. By connecting world-state evolution with experience generation in a continuous interaction loop, the framework offers an approach to building interactive worlds that retain their history, evolve through action, and persist beyond any individual observer.