Policy and tools evolve together for better long video understanding
CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding
Computer Vision and Pattern Recognition
Summary
Working with very long videos is tricky because you need to find important moments without looking at everything closely, which takes a lot of time. The authors propose a way for an AI to improve both its strategy (policy) and the tools it uses at the same time by learning from its own past attempts. This means the AI can get better at spotting events over long videos while using less computing power. Their tests show that this method is more accurate and efficient across different videos and tasks, and the improved skills can also help understand other kinds of long videos better.
What this means in practice
- •For video analytics teams: Use evolving policies and tools to improve event detection accuracy in ultra-long surveillance or sports videos while reducing computational cost.
- •For video content platform developers: Enhance video search and question answering features by enabling models to better coordinate image and video information over long durations without separate planning systems.$Commercial implications: Supports commercial video platforms in providing advanced interactive search and QA functionalities that rely on efficient long video understanding.
Authors
Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen
Abstract
Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.