Vision language models reduce cost by pruning visual tokens smartly
Vision Token Pruning via Query-Vision Interaction Decomposition
Machine LearningComputer Vision and Pattern Recognition
Summary
Large models that understand images plus text are slow because they look at many small pieces of an image. The authors found that the way text and image pieces interact can be simplified into a few key patterns. Using this insight, they created a method to pick only the important image pieces without extra training. This makes the model faster while keeping accuracy, especially when removing many pieces.
What this means in practice
- •For mobile app developers: Make vision-language apps run faster on phones by cutting down unnecessary image processing based on text inputs.
- •For video analytics teams: Improve speed and efficiency in video content analysis by selecting only the most relevant visual information guided by textual queries.
Authors
Harshithanjani Athi, Sravan Kumar Ankireddy, Jianzhong Charlie Zhang, Hyeji Kim
Abstract
Large vision-language models (VLMs), which process both visual inputs and text queries, incur high inference costs due to the large number of visual tokens, making visual token pruning a natural approach for improving efficiency. Recent work has leveraged query information to guide token selection, often through heuristics derived from attention patterns or token dynamics. However, these approaches do not explicitly exploit the underlying structure of query-vision interactions. We address this gap by observing that the query-vision interaction matrix, naturally induced by the query and key projections already present in transformer attention, is effectively low-rank, with a small number of dominant latent interaction modes capturing query-relevant semantics. Motivated by this observation, we propose Query-Vision Decomposition (QViD), a novel training-free query-aware visual token pruning method that exploits this structure. Extensive experiments on both image and video understanding benchmarks show that QViD consistently improves performance under matched token budgets, with the largest gains appearing under aggressive compression regimes.