Agents using vision language models can search videos as well as humans
Can Agents Win the Video Browser Showdown?
Multimedia
Summary
Searching through large collections of videos is usually a back-and-forth job where people say what they want, look at results, and then adjust their search. This paper looks at whether smart computer agents that use vision and language understanding can do this searching by themselves from the very start. The challenge is that these agents don’t know all the videos beforehand, so they must work with a search system that indexes videos and helps them ask good queries. The agents then look closely at top video results to decide what to do next. The researchers found that these agents can often find videos just as well as expert human searchers, working fully on their own.
video retrievalvision language modelsinteractive searchautonomous agentssearch intentquery refinementvideo indexingiterative searchagentic system
Authors
Bastian Jäckl, Zuzana Vopálková, Daniel A. Keim, Jakub Lokoč
Abstract
Searching large video collections is typically an interactive process in which users play two roles. First, they hold the search intent: the underlying goal that determines what content they seek and why. Second, users must operationalize this intent through an iterative search loop. Users translate their intent into queries, browse the retrieved candidates, and refine their queries based on the results. In this paper, we investigate the capabilities of modern Vision Language Models (VLM) and agentic approaches to reach search goals interactively and fully autonomously. Specifically, we study whether a provided initial specification of a search goal might be sufficient to solve traditionally interactive search tasks with an agentic system. Provided that the involved VLMs are not aware of the whole large video dataset in advance, the key challenge lies in the effective combination of an existing interactive video search system and a smart VLM agent controlling the system. While the search system provides indexing and efficient querying, the VLM-based agents analyze top-ranked items and make decisions about next actions. Our results show that modern agents can autonomously operate interactive video retrieval systems to solve many search tasks from an initial intent description, achieving performance competitive with strong historical expert-operated systems in several settings.