VidHarness automates video agents for cheaper long video analysis
VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding
Computer Vision and Pattern Recognition
Summary
Looking at every moment in hour-long videos to answer questions is too slow and costly. The authors created VidHarness, a system that automatically designs smart programs called harnesses to watch only important video parts. It uses a search method to find the best way to watch videos efficiently, saving processing time and improving accuracy. This approach works better than expert-made methods and adapts to different budgets for analyzing video frames.
What this means in practice
- •For video processing teams: Cut video analysis costs and improve accuracy by using automatically optimized harnesses that watch fewer frames selected for each question.
- •For security monitoring operators: Monitor hours-long security footage more efficiently by routing questions to specialized harnesses tailored to available computational budgets.
Authors
Susan Liang, Jianmin Wu, Daxiang Dong
Abstract
Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to $11.2$ points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.