Vision language model pruning boosts speed retaining most performance
TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models look at images and answer questions, but they can be slow because they analyze many parts of an image. The authors show that if the model smartly keeps the important image parts related to the question while ignoring unneeded ones, it speeds up without losing much accuracy. They found it helps to combine how much an image part stands out with how relevant it is to the question. Their new method, called TReVS, cuts down the image parts by over 90% while keeping almost all of the model's performance.
What this means in practice
- •For mobile app developers: Create faster vision-language apps that respond quickly by pruning irrelevant image tokens while keeping key information.
- •For cloud ai service teams: Reduce cost and latency in cloud-based vision-language services by efficiently pruning visual tokens without harming answer quality.
Authors
Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li
Abstract
Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visual tokens. Recent visual token pruning methods increasingly follow a two-stage paradigm: they first remove visually redundant tokens after the vision encoder and then discard tokens irrelevant to the textual query within the Large Language Model (LLM). However, since the first stage typically relies solely on vision-encoder saliency, it may prematurely eliminate query-relevant tokens, depriving the subsequent text-guided stage of critical visual evidence. Our empirical analysis shows that incorporating query guidance into first-stage pruning better preserves task-relevant evidence and consistently improves performance over vision-only saliency-based pruning. We further find that high-variance attention heads are more sensitive to the textual query and yield more discriminative text-to-vision attention signals for second-stage pruning. Motivated by these findings, we propose TReVS, a training-free framework that combines textual relevance with vision-encoder saliency for pre-LLM pruning and leverages high-variance attention heads to remove task-irrelevant tokens at shallow-to-intermediate layers of the LLM. On LLaVA-1.5-7B, TReVS retains 92.8% of the unpruned baseline performance while pruning 94.4% of visual tokens, outperforming prior state-of-the-art methods.