Video token compression speeds up vision language model inference
VETO: Video Efficient Token Optimization for Vision Language Models
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Processing long videos with vision-language models is very slow because the models need to look at many pieces of a video frame by frame, which takes a lot of computing power. The authors present VETO, a method that reduces this workload by smartly grouping similar parts within each frame and merging similar frames across time. This two-step approach makes the video analysis faster without losing accuracy. Tests show VETO can speed up video understanding by almost half while keeping or improving results.
What this means in practice
- •For video analytics engineers: Speed up processing in video analysis systems by reducing the number of visual tokens without losing accuracy for long videos.
- •For ai infrastructure developers: Improve the efficiency of vision-language model inference pipelines by integrating a dual-axis token compression approach to handle video data cost-effectively.
Authors
Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen, Winston H. Hsu
Abstract
Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.