AI summaryⓘ
The authors propose OmniAgent, a new way to understand long videos by actively choosing which parts to watch instead of watching everything uniformly. It treats video understanding as a step-by-step process where the agent observes, thinks, and acts, storing important information in text form to avoid heavy computation over the entire video. They developed special training methods to help the agent learn when to focus attention and how to improve based on uncertainty in its decisions. Tests on multiple benchmarks show that OmniAgent performs very well, even beating larger models on some tasks. This shows that actively selecting what to watch can make video understanding more efficient and accurate.
long video understandingactive perceptionPOMDPreinforcement learningagentic fine-tuningaudio-visual cuestextual memoryuncertainty-based credit assignmentbenchmark evaluationLVBench
Authors
Zhenghao Xing, Ruiyang Xu, Yuxuan Wang, Jinzheng He, Ziyang Ma, Qize Yang, Yunfei Chu, Jin Xu, Junyang Lin, Chi-Wing Fu, Pheng-Ann Heng
Abstract
Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).