Video token compression improves efficiency for large language models
Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline
Computer Vision and Pattern Recognition
Summary
Video language models understand videos but slow down when handling lots of visual details. The authors studied how much simpler methods could compress video data without losing important information. They introduced a straightforward way to group and average visual features across frames that doesn’t require extra training. Their method works well compared to other complex approaches, especially when allowed to keep very few details. They found that keeping the overall visual feature spread is key to maintaining good understanding.
What this means in practice
- •For mobile app developers: Reduce computation cost when running video language models on devices with limited resources by compressing video input tokens without much accuracy loss.
- •For video analytics teams: Improve real-time video content understanding efficiency by adopting simple clustering-based compression methods to reduce input data volume.
Authors
Xiao Zhang, Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Shichao Kan
Abstract
Video Large Language Models (Video LLMs) have achieved remarkable progress in video understanding, but their inference efficiency is constrained by the large number of visual tokens produced by long videos. Recent video token compression methods increasingly introduce sophisticated strategies for token selection, pruning, and merging. This raises a fundamental question: how much of compression performance can be obtained by simply preserving the structure encoded in the visual representations? We investigate this question with SimpleCluster, a simple and training-free baseline that performs position-aware cross-frame clustering in the visual feature space and represents each cluster using the mean of its original visual features. Extensive experiments across four video understanding benchmarks and three representative Video LLMs show that SimpleCluster achieves competitive or superior performance over recent compression methods across a wide range of token retention ratios, with particularly strong robustness under extremely low retention rates (e.g., 1%). To understand this behavior, we analyze the feature space preserved by different compression methods in terms of local approximation fidelity and global coverage. The results show that stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage. These findings highlight feature-space preservation as an important consideration for video token compression under highly constrained token budgets. Our code is available at https://github.com/xiaozhang79/SimpleCluster.