Wizard enables real time control of game worlds with diverse characters
Wizard: Manipulate Any World in Real Time via Agentic Control
Computer Vision and Pattern Recognition
Summary
Video games often have limited ways to control characters, and new game settings can be hard to manage. This paper presents Wizard, a system that lets players control characters' movements precisely while creating scenes that the system has never seen before. The authors built a large collection of scenes and characters and used smart representations of character shapes and movements to guide video creation. Wizard also keeps track of game states so players can customize their virtual worlds in real time without losing quality or consistency.
What this means in practice
- •For video game developers: Create virtual worlds where players can control diverse characters in real time with precise movements and novel scenes without predefining all actions.
- •For animation studios: Generate consistent character animations in new environments on the fly by conditioning on structured skeleton and geometry data.
Authors
Yuzhe Wu, Zijun Lin, Yeying Jin
Abstract
Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.