BtsPruner cuts visual tokens to speed up large vision language models
BtsPruner: Role-Aware Visual Token Pruning for Large Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Processing many image parts in big vision-language AI models is slow and uses a lot of memory. The authors found these parts change roles as they are analyzed, from showing fine details to representing the whole image. They created BtsPruner, a method that saves important parts at each stage without extra training, keeping a mix of details and summaries. This method keeps most of the model’s accuracy while using far fewer image parts, making the AI run much faster.
What this means in practice
- •For ai application developers: Reduce visual token inputs in large vision-language models to speed up real-time image understanding applications without retraining.
- •For edge device engineers: Lower the computational and memory needs of vision-language inference on resource-limited devices by pruning visual tokens while preserving accuracy.
Authors
Zihao Qi, Zhibo Zhang, Miaoxin Zhang, Jincheng Cai, Zengqiang Yan
Abstract
Large Vision-Language Models (LVLMs) often process hundreds or thousands of visual tokens, resulting in substantial computational and memory overhead during inference. Existing training-free pruning methods typically estimate token importance using language-model attention, visual attention, or feature similarity. However, these methods treat visual tokens with uniform criteria and overlook that tokens may serve different roles during visual encoding. In this work, we investigate the evolution of visual token roles in LVLMs and observe that tokens progressively transition from local detail carriers to globally referenced representatives as encoding proceeds. Based on this observation, we propose BtsPruner, a training-free role-aware visual token pruning framework that preserves complementary token roles under a fixed budget. Specifically, BtsPruner first identifies the token-role transition point based on the evolution of attention concentration. It then discovers globally referenced representatives from later encoding stages and recursively retrieves local detail tokens from earlier stages to recover complementary fine-grained evidence. Extensive experiments across LLaVA-1.5, LLaVA-NeXT, Qwen-VL, InternVL, and CogVLM demonstrate that BtsPruner achieves superior performance retention under aggressive token reduction. On LLaVA-1.5-7B, BtsPruner preserves 91.6% of the dense model's performance using only 32 of 576 visual tokens (5.6%). In the high-resolution LLaVA-NeXT-7B setting, it achieves up to a 5.95x CUDA-latency speedup.