Video harness evolves from unlabeled videos for better video understanding
VideoEvoHarness: Self Evolving Agent Harness from Unlabeled Videos for Video Understanding
Computer Vision and Pattern Recognition
Summary
Understanding videos automatically is hard and usually needs lots of labeled examples and detailed instructions to improve. The authors present a way for a video analysis system to teach itself by creating and answering questions about videos even when no labels are given. Their system improves over time by using feedback on its answers to evolve both the questions it asks and the ways it solves them. This method works without changing the main video model itself and shows better accuracy on tests with new videos not seen during learning.
What this means in practice
- •For video analytics engineers: Improve video analysis tools by automatically refining question and reasoning strategies using unlabeled videos without extra manual labeling.
- •For ai tool developers: Develop video agents that self-optimize their problem-solving workflows and tool usage based on performance feedback from unlabeled video data.
Authors
Weixin Xu, Yifan Liu, Yi Wang, Mingfei Rong, Zhenyu Yang, Zhisheng Zhong, Shengsheng Qian, Changsheng Xu
Abstract
Video understanding agents rely on a harness to acquire evidence, use tools, and reason, but improving it requires substantial effort. Recent self evolution methods seek to optimize harnesses through execution feedback to reduce human intervention. However, existing methods often rely on annotated video questions and answers or focus on updating model parameters, leaving harness evolution from unlabeled videos underexplored. VideoEvoHarness jointly evolves a Questioner and a Solver harness from unlabeled videos with frozen model parameters. The Questioner constructs questions and computes reference answers directly from video memory records. Question quality assessments and Solver execution trajectories guide question generation policy revisions. The Solver supports evolving skills, strategies, hyperparameters, and tools using execution feedback. Executable strategies define workflows in code, reducing repeated interpretation of natural language instructions. Each round, candidate harnesses are compared on identical validation questions, with validated improvements retained for later use. Experiments on benchmarks and a dedicated benchmark question set, using videos disjoint from evolution videos, show higher overall accuracy than initially and transfer gains across backbones. Overall accuracy on this set also improves progressively across rounds. This supports our foundational framework for harness evolution from unlabeled videos, opening a promising direction for self evolving video agents. Code is available at https://github.com/Wei-Xin-Xu/VideoEvoHarness.