Video large language models cut tokens for faster understanding
GleanVID: Complementary Token Selection for Efficient Video Large Language Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Video language models can understand videos very well but are slow because they process many visual parts called tokens. The authors see token selection as a step-by-step process to keep the most useful and least repetitive parts across video frames. They created GleanVID, a way to speed up video understanding without extra training, by picking tokens based on how different and informative they are over time. Their tests show GleanVID keeps almost all understanding ability while cutting the number of tokens by 75%, making processing much faster.
What this means in practice
- •For video app developers: Reduce video processing delay by selecting fewer but more informative video features during analysis with video large language models.
- •For streaming platform engineers: Cut computational cost on live video understanding by efficiently prioritizing key visual data across frames using a budgeted token approach.
Authors
Shuo Yang, Changbai Li, Rui Tang, Xinyu Zhao, Linlin Yang, Baochang Zhang
Abstract
Video Large Language Models (VideoLLMs) have achieved strong video understanding capabilities but incur substantial inference overhead due to the large number of visual tokens. Existing VideoLLM token compression methods largely rely on selection-independent scoring, overlooking cross-frame complementarity and consequently retaining redundant evidence across frames. Instead, we view video token selection as a progressive evidence accumulation process. It aims to retain visual evidence that is individually informative and collectively complementary under a limited token budget. Building on this insight, we introduce GleanVID, a training-free inference acceleration framework for VideoLLMs. Specifically, GleanVID first allocates the global token budget across frames according to temporal novelty and then selects tokens by jointly considering local representativeness and subspace complementarity, thereby preserving richer and less redundant visual evidence. Extensive experiments across diverse VideoLLMs and benchmarks demonstrate that GleanVID consistently achieves state-of-the-art performance. Notably, with only 25% of visual tokens, GleanVID preserves 98.6% of Qwen3-VL's original performance while reducing its prefill latency by 44.7%. On LLaVA-OV-7B, GleanVID at a 25% retention ratio even slightly surpasses the original model.