Modern AI agents autonomously search large video collections effectively
Can Agents Win the Video Browser Showdown?
Multimedia
Summary
Searching through many videos usually requires people to guess what to look for, look at results, and try again. This paper shows that smart AI agents can do this job by themselves if they get a clear goal at the start. These agents use vision language models to look at videos and decide what to do next without human help. The researchers found that this approach works well and can compete with expert human searchers in some cases.
What this means in practice
- •For video platform developers: Create systems that let AI agents autonomously find relevant videos based on initial search goals without user interaction.$Commercial implications: Enables video platform companies to offer smarter search tools that reduce user effort and improve discovery, making it marketable as an advanced feature.
- •For media monitoring teams: Use autonomous search agents to quickly scan large video archives for content matching specific investigation goals without manual searching.
Authors
Bastian Jäckl, Zuzana Vopálková, Daniel A. Keim, Jakub Lokoč
Abstract
Searching large video collections is typically an interactive process in which users play two roles. First, they hold the search intent: the underlying goal that determines what content they seek and why. Second, users must operationalize this intent through an iterative search loop. Users translate their intent into queries, browse the retrieved candidates, and refine their queries based on the results. In this paper, we investigate the capabilities of modern Vision Language Models (VLM) and agentic approaches to reach search goals interactively and fully autonomously. Specifically, we study whether a provided initial specification of a search goal might be sufficient to solve traditionally interactive search tasks with an agentic system. Provided that the involved VLMs are not aware of the whole large video dataset in advance, the key challenge lies in the effective combination of an existing interactive video search system and a smart VLM agent controlling the system. While the search system provides indexing and efficient querying, the VLM-based agents analyze top-ranked items and make decisions about next actions. Our results show that modern agents can autonomously operate interactive video retrieval systems to solve many search tasks from an initial intent description, achieving performance competitive with strong historical expert-operated systems in several settings.