Video generation agents improve quality by using external tools
VideoGen-Agent: Reinforcing Video Generation Agents
Computer Vision and Pattern Recognition
Summary
Generating videos from text is hard because it needs detailed knowledge, consistent scenes, and correct sequences. The authors developed VideoGen-Agent, which acts like a video-making assistant that uses different tools step-by-step to create better videos. It learns from examples and improves through trial and error, becoming better at picking the right tools for each part of the video. This agent was tested on a benchmark with 600 challenges and made videos that people liked much more than the original system.
What this means in practice
- •For video content creators: Create complex videos that require multiple scenes or consistent characters using an agent that automatically combines various generation tools.
- •For animation studios: Improve video generation workflows by integrating an agent that learns to apply specialized tools for physical consistency and scene composition.
- •For advertising agencies: Generate high-quality, multi-scene promotional videos with preserved branding elements by using an intelligent tool-coordinating agent.$Commercial implications: This enables sale of automated video production services that maintain brand consistency and complex scene design.
Authors
Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang, Zihao Lin, Kaituo Feng, Suozhi Huang, Xiangyi Li, Yu Li, Chunyuan Li, Shilong Liu, Mengdi Wang
Abstract
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.