Multi-agent planning improves physics in text to video generation
PhyEvolve: Multi-Agent State-Evolution Planning for Physics-Aware Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos from text prompts often misses how physical objects change and move over time. The authors introduce PhyEvolve, which uses three cooperating agents to plan out how objects and events evolve physically in a video. This planning improves how the video matches the prompt, capturing cause-and-effect changes more clearly. Their system works with existing video generators without changing them, just by rewriting prompts, and outperforms previous methods in tests.
What this means in practice
- •For video content creators: Generate videos from text prompts that better depict realistic physical changes over time using enhanced prompt planning.
- •For game developers: Create more physically consistent animated sequences in games by integrating planned physical state changes inferred from natural-language descriptions.
Authors
Yang-Hao Zhou, Lixing Tan, Zhiqi Li, Guang Yang, Xiao Yang
Abstract
Recent advances in text-to-video generation have substantially improved visual fidelity, yet existing models still struggle to understand the physical evolution implicit in user prompts, making it difficult to generate videos with continuous physical state evolution. Here, we propose PhyEvolve, a multi-agent framework that enhances natural-language prompts through structured state-evolution planning. PhyEvolve consists of three agents: a Director, a Physical Evolver, and a Composer. Given a user prompt and an optional first frame, the Director grounds the scene and disentangles physical events from observation changes. The Physical Evolver uses scene evidence to identify physical mechanisms then constructs causally linked state transitions that unfold throughout the video, whether visible or not. The Composer utilizes these transitions to enhance the original prompts. Without retraining or model modification, PhyEvolve works directly with existing generators through prompt replacement. Extensive experiments on four major benchmarks show that PhyEvolve achieves state-of-the-art performance, outperforming the cutting-edge prior agentic baseline by 7.9 percentage points on average. These results demonstrate that explicit causal state-evolution planning improves physical consistency and state continuity in generated videos through natural-language prompt enhancement.