Video evolve improves event localization in videos from language queries
VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Video temporal grounding means finding the exact time an event happens in a video based on a spoken or written description. The authors present VideoEvolve, a way to automatically improve how an agent uses video and language understanding models to find these events. Their method changes the steps and instructions the agent uses, testing and keeping the best changes to get better results. They showed this approach improves performance on several tests, especially by refining the instructions the agent follows.
What this means in practice
- •For video indexing teams: Improve automated event detection in video content by evolving agent workflows to better interpret natural-language queries.
- •For video content moderation teams: Enhance temporal localization of events in video streams to more accurately identify and act on specific content segments.
Authors
Bingjun Luo, Yuhuan Fan, Jialin Guo, Siqi Li
Abstract
Video temporal grounding aims to localize events in videos from natural-language queries. For agents built around frozen video-language models, the harness determines how queries guide temporal predictions and how those predictions are refined. Manually refining these harnesses requires diagnosing grounding failures and coordinating changes to both agent workflows and instructions. We introduce VideoEvolve, a framework that automatically evolves agent harnesses for video temporal grounding. VideoEvolve uses a Cloze-Structured Harness Representation that preserves stage interfaces while leaving agent workflows and instructions open to evolution. Branch-Guided Harness Evolution preserves promising code branches for continued refinement, using execution feedback to guide local edits and validation to determine which improvements are carried forward. Experiments demonstrate improved grounding performance across multiple benchmarks. Component analyses identify instruction refinement as a consistent source of gains, while the benefits of evolved code vary across evaluation settings. Together, these results support automated harness evolution as an effective approach to improving video temporal grounding. Code is available at https://github.com/bingjunluo/VideoEvolve .