AI summaryⓘ
The authors created OmniReasoner, a system that helps large language models understand long videos with sound better by first looking at a quick, low-detail preview and then zooming in on important parts for closer examination. This method saves computing power by focusing on key moments instead of processing the whole video in high detail. They also developed a method called TimeAnchor to keep the timings correct when switching between low and high detail views. To train their system without expensive manual labeling, they made a tool that generates training data by editing videos. Their tests show this approach improves how well the system answers questions and identifies when things happen in videos.
Omnimodal LLMsaudio-video reasoningtool-use learningsupervised fine-tuningreinforcement learningtemporal groundingTimeAnchorvideo samplingdata augmentationzoom-in tool
Authors
Yu Chen, Caorui Li, Ziyu Xiong, Yidong Wang, Mingqi Gao, Shuman Liu, Biao Liu, Chunfeng Yang, Anxiang Zeng, Haibo Zhang, Chaofan Chen
Abstract
Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call -- a sparse global preview and a denser local clip -- we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by video editing and composition. Experiments across omnimodal and video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.