WorldAgent builds and checks complex 3D worlds from language prompts
WorldAgent: Verification-Guided Agentic Physical World Construction
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Creating detailed 3D physical worlds from just a short language description is very hard because it involves many objects, their arrangement, and how they interact physically. The authors present WorldAgent, a system that automatically builds these worlds step-by-step, checks if everything makes sense physically, and fixes problems without needing the user to intervene repeatedly. The system also allows users to later inspect and adjust the created worlds. Their tests show WorldAgent performs better than similar systems on many measures and users prefer it overall.
What this means in practice
- •For game developers: Automatically create detailed and physically consistent 3D game scenes from simple text descriptions, speeding up level design.$Commercial implications: Enables commercial tools for rapid content generation that reduces manual scene building costs and time.
- •For virtual reality creators: Generate realistic and interactive virtual worlds from natural language commands for immersive experiences without extensive manual setup.
Authors
Caoliwen Wang, Mengdi Wang, Yige Chen, Zejia Wu, Bowen Huang, Siyuan Chen, Guanxiong Chen, Lifu Wei, Heng Zhang, Qinghai Zhang, Yin Yang, Guandao Yang, Shiying Xiong, Peng Wang, Chenfanfu Jiang, Peter Yichen Chen
Abstract
Constructing complex physical worlds from language requires coordinating extensive 3D environments, detailed structures and objects at different spatial scales, and interacting physical processes under both stated goals and implicit physical constraints. We present WorldAgent, an agentic framework for verification-guided physical world construction from a single natural-language prompt, without iterative user debugging. A world construction layer expands the prompt into a structured world specification and uses physical knowledge to build scenes and run numerical simulations. After every step, a verification layer inspects scene geometry and simulation states alongside rendered views. Failed checks guide automatic revisions to the specification and re-execution of the affected steps. Accepted worlds pass the required checks and remain editable for further inspection and resimulation. We introduce AgenticSimBench, on which WorldAgent achieves the best scores among the evaluated agent-based methods on five of seven metrics. In a 26-participant user study, it receives the highest mean ratings across all four criteria.