Text2Sim generates editable physics simulations from text requests
Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise
GraphicsArtificial Intelligence
Summary
Creating detailed physical simulations is usually hard and takes a lot of time because many parts like objects, movements, and rules need to work together correctly. The authors designed Text2Sim, a tool that reads a text description and automatically builds a complete, working simulation that can be edited. It uses a special system that plans the simulation, creates assets, checks for errors, and fixes problems with expert knowledge. When tested, Text2Sim produced simulations that looked and behaved better than other recent methods, and more people preferred its results.
What this means in practice
- •For simulation developers: Automatically create and edit diverse physics-based simulation scenarios from textual descriptions to speed up content creation.
- •For game developers: Generate realistic and customizable physics simulations for games without manual asset and parameter tuning.$Commercial implications: Enables production of complex simulations efficiently, lowering costs and time for game studios producing physics-heavy content.
Authors
Xiaoyu Xiong, Tsun-Hsuan Wang, Yi-Ling Qiao, Tao Du, Minchen Li
Abstract
Creating diverse physical simulations remains labor-intensive because assets, layout, physical parameters, motion, control, and rendering must be designed and debugged jointly. We present Text2Sim, a simulation-specialized agentic pipeline that converts a text-only request into an executable, editable dynamic case. Built on Genesis, Text2Sim uses a hierarchical agentic structure that combines a Planner with specialized Writers, asset-generation tools, and an independent Critic. Compact skills (Debug Cards) distilled from graphics demonstrations provide role-specific physical guidance for execution-based repair. We evaluate physical quality, visual quality, and human preference on 42 held-out prompts spanning rigid, articulated, deformable, and cloth phenomena, with a paper-level split between experience construction and evaluation. We design automatic physical and visual scorers to evaluate the quality of the results, and Text2Sim achieves higher scores than all four state-of-the-art baselines on both metrics. In blinded user studies with these baselines, significantly more participants prefer Text2Sim than prefer the baselines, which is consistent with the results from our automatic scorers. The pipeline also supports a broad range of downstream applications; we select dataset construction and extension to multimodal input as two representative examples. We will release the code, the Debug Card library, and a dataset of generated cases, each pairing the text prompt and rendered video with the executable program, assets, physical parameters, controls, and recorded states.