AgentDreamer guides 3D physics simulations with text instructions
AgentDreamer: Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation
Computer Vision and Pattern Recognition
Summary
Simulating physical motion based on written instructions is difficult because machines must understand language and physical laws at the same time. The paper presents AgentDreamer, a method where an AI agent first turns text instructions into detailed motion plans in 3D space. Then a physics simulator adjusts these plans to make sure the movements are realistic and obey physical rules. This keeps the connection between language and actual motion clear and grounded in physics. The authors also introduce a new way to measure how well the motion matches the instructions.
What this means in practice
- •For game developers: Generate physically realistic character or object movements in games directly from narrative text descriptions.$Commercial implications: Enables new game mechanics and interactions by translating player text commands into precise simulated actions.
- •For visual effects artists: Create detailed and physically plausible animations for scenes by guiding simulations with natural language instructions.
Authors
Mijeong Kim, Suho Park, Junha Park, Bohyung Han
Abstract
We propose AgentDreamer, an agent-guided optimization framework that aligns text instructions with simulated motion through explicit 3D motion planning. Given a text instruction and scene geometry, an off-the-shelf agent translates the instruction into a structured semantic motion prior, specifying what should move, in which direction, and in what temporal order. The agent represents this prior using parametric functions over a series of temporal chunks, yielding dense 3D trajectories. We then optimize the parameters of a differentiable MPM simulator such that its rollout best matches the prior while remaining constrained by the scene dynamics. This allows the agent to provide semantic guidance from language, while the simulator grounds it in physically feasible motion. We further introduce Motion-IG, an information-theoretic metric for text-motion alignment based on masked text recovery. By comparing the likelihood of recovering masked motion-related content with and without the simulated motion, Motion-IG measures the information contributed by the motion beyond that available from the remaining text and scene context. Experiments show that AgentDreamer improves text alignment over video-based guidance, producing simulations that more closely follow the requested physical behavior.