Video understanding improved by new system for harness evolution
VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Video understanding systems rely on harnesses that organize how information is gathered and used. The authors study how to automatically improve these harnesses despite noisy and limited feedback. They introduce a new framework, VideoHarness-SRSI, that carefully controls how feedback guides improvements to avoid common pitfalls. Their system outperforms previous methods across several tests, showing that the way the system searches for improvements is as important as how well it evaluates itself.
What this means in practice
- •For video analytics teams: Improve automated video analysis tools by evolving better information processing strategies around fixed vision-language models.
- •For ai system builders: Design more reliable recursive self-improvement methods for complex agent systems using evidence-governed search frameworks.
Authors
Zihao Lin, Haoyu Dong, Haobo Xu, Chi-Chih Chang, Xuying Ning, Yinglong Xia, Shuang Yang, Zhuokai Zhao, Xinying Song, Dongfang Liu, Lifu Huang, Yang Song, Qifan Wang
Abstract
LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reasoning, and memory. Recursive harness evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reused feedback remains unclear. We study this question in agentic video understanding, where rich multimodal evidence and coupled mechanisms make harness optimization challenging. We first introduce VideoHarness-BRSI, a recursive self-improvement framework that evolves video harnesses around a frozen vision-language model. Analyzing over 4,000 evolved candidates, we find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. We trace all three to a common root: the same imperfect validation evidence drives both harness revisions and the proposer's search strategy. We therefore propose VideoHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe-race branches before factorial composition and joint refinement. Across four video understanding benchmarks, VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative to VideoHarness-BRSI and reduces the mean validation-test gap by 37.9%. Reliable harness evolution thus depends on how search is organized and informed, not only on how well it optimizes validation performance.