VideoMM speeds up long video understanding with adaptive token focus
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionMultimedia
Summary
Understanding long videos with AI is usually slow and expensive because the models have to look at lots of visual details all at once. The authors found that not all details are equally important at every step; early on, only a rough look is needed to find important parts. VideoMM first scans a low-detail version of the video to pick out key regions and then only zooms in on those parts in high detail. This approach makes video understanding much faster and more accurate than previous methods.
What this means in practice
- •For video platform engineers: Process long videos faster and more accurately by selectively analyzing key regions with high detail only when needed.
- •For security monitoring teams: Improve real-time video analysis by reducing computational load while maintaining detailed insights on important events.
Authors
Haoyu Guo, Yuan Feng, Junlin Lv, Mingjun Xiao, S Kevin Zhou, Xike Xie
Abstract
Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates context windows and incurs prohibitive costs. Current solutions predominantly rely on auxiliary models for token reduction but face a fundamental dilemma: lightweight encoder-driven approaches often overlook critical semantic information, whereas heavyweight MLLM-driven reduction negates the efficiency gains. {In this work, we identify a more fundamental inefficiency underlying this dilemma: while fine-grained visual details are essential for detailed understanding, they are largely redundant for the preliminary task of selecting semantically relevant regions. } Motivated by this, we introduce \textbf{VideoMM}, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity. Specifically, our framework {decouples selection from reasoning} by executing semantic filtering on a cost-effective \textit{Macro Proxy} (derived from downscaled frames), and projecting the selected regions onto high-fidelity \textit{Micro Tokens} for detailed understanding only when necessary. Extensive evaluations show that VideoMM significantly outperforms existing solutions. It achieves a 6.13$\times$ speedup and a 7.4\% accuracy gain over full-context baselines on LongVideoBench, and further accelerates inference by 2.73$\times$ over current leading methods, establishing a highly scalable paradigm for long-video understanding. Our code is available at: https://github.com/adfh917k/VideoMM.