Leap retrieves key evidence for long audio video questions
LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Long audio and video recordings make it hard for AI to understand details because either it tries to process everything and runs out of memory or it compresses too much and loses important clues. The authors created LEAP, which breaks recordings into small parts and picks the most relevant pieces before answering questions, so it uses less space but keeps important sounds and images. LEAP can work on live streams and was tested to perform better than previous models on several benchmarks.
What this means in practice
- •For media analysis teams: Improve question answering over hours of video footage by efficiently selecting relevant sections for detailed analysis.
- •For video content platform developers: Enable real-time question answering on live audio-video streams without extra training for streaming use cases.
Authors
Juyi Lin, Zhiqiang Lao, Jiali Cui, Lin Zhao, Pu Zhao, Dichang Zhang, Arman Akbari, Yu Qi, Xinru Jiang, Yanzhi Wang, Heather Yu, Liang Peng
Abstract
Hour-scale audio-visual question answering is constrained by a context dilemma: dense whole-recording encoding rapidly exhausts context limits, whereas uniform temporal compression severely dilutes fine-grained acoustic and visual evidence. We introduce LEAP, a framework where the model retrieves its own evidence without placing the whole recording in one context. LEAP divides a recording into fixed-duration blocks, applying a lightweight localization pass to each block to score short candidate windows. The highest-ranked windows are pooled and re-encoded in a single bounded answer pass. Consequently, the answer input and peak context remain independent of the recording duration. By decoupling evidence localization from reasoning, our framework can localize candidate temporal windows over pre-computed transcripts without decoding media frames, while preserving fine-grained visual and non-speech evidence by routing the final answering pass over raw audio-visual streams. LEAP trains both stages: a localization LoRA improves the selected windows, and an answer LoRA improves the answers read from the same windows. The block grid natively supports causal queries, enabling LEAP to support streaming inference without streaming-specific training. Across several AVQA benchmarks, LEAP improves over the Qwen3-Omni-30B-A3B baseline by 4.5-16.8%, and transfers to a second omni-modal backbone, MiniCPM-o 4.5, surpassing its published results by 3.1-13.0%.