Code2video improves video generation with proxy guidance and adaptive control
Coda-DaVinci: Reimagining Video Generation with Code2Video
Computer Vision and Pattern Recognition
Summary
Generating videos that look realistic and follow directions well is challenging. The authors study how using simplified 3D models, called proxies, can guide video generation to control how things move and look. They find that different levels of detail in these proxies trade off between natural motion and consistency, and that adding physics simulations and matching prompts helps. To put these insights into practice, they build Coda-DaVinci, a system that adjusts how it creates videos depending on the task, improving the video quality and accuracy of following instructions.
What this means in practice
- •For animation studios: Create videos with explicit control over object motions and scene consistency using proxy-guided generation combined with physics simulation.$Commercial implications: Enables production of higher-quality videos with tailored dynamics and visual realism for animation professionals.
- •For game developers: Generate consistent story-driven video sequences that adhere to scene settings and physics for in-game cutscenes or storytelling.
Authors
Yiyan Ji, Qixun Wang, Yang Shi, Zhuoran Zhang, Yifan Xu, Tengfei Liu, Yuqi Tang, Xiaohan Zhang, Shujie Li, Jiafu Tang, Xuanyu Zhu, Xinyu Liu, Yan He, Yushuo Guan, Zihao Wang, Xianghua Ying, Shixuan Liu, Haotian Wang, Wenjing Yang, Zhaoxiang Zhang, Yuanxing Zhang, Pengfei Wan, Jiaheng Liu
Abstract
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.