Video agents improve understanding by revising their observation methods
Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Video understanding systems learn by watching videos and following a set method for what to look at and analyze. The authors created a way for these systems to improve themselves by changing their own methods using their language skills, allowing them to gather better evidence when videos don’t make sense. This self-improvement also balances the need for accuracy and the amount of video watched, so the system can be more efficient. Tests showed the improved systems were more accurate and used less video than before, competing well with other methods.
What this means in practice
- •For video analytics teams: Improve automated video analysis accuracy while reducing the amount of video data processed, cutting costs without sacrificing performance.
- •For security monitoring operators: Implement adaptive video review agents that refine their observation strategies over time to detect events more accurately and efficiently.
Authors
Bingjun Luo, Jialin Guo, Siqi Li
Abstract
Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement. We introduce Video-RSI, a framework for recursive self-improvement in which a video understanding agent uses its own language model to revise its harness. Through active video investigation, the model revisits the original training videos to test competing failure explanations with additional observations, grounding proposed changes in evidence beyond the existing trace. Cost-aware harness evolution turns these diagnoses into reusable revisions and determines which revisions to retain by considering both answer accuracy and visual cost. Across our evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames and achieves competitive accuracy-efficiency trade-offs against existing video understanding agents. These results demonstrate the potential for video understanding agents to improve their own evidence acquisition and use through harness evolution. Code is available at https://github.com/bingjunluo/Video-RSI .