Visual token pruning improves speed and efficiency in large vision language models
ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs
Computer Vision and Pattern Recognition
Summary
Large vision-language models use many small pieces called visual tokens to understand images, but this can make them slow. The authors propose a way to pick just the most important visual tokens to keep, based on how well they cover the image and connect to the text. Their method, called ACPruner, works without retraining the model and speeds up processing while still keeping the model’s accuracy. They tested it on several big models and found it keeps performance high while making inference faster.
What this means in practice
- •For machine learning engineers: Speed up the inference of vision-language models by efficiently selecting key image tokens without retraining the model.
- •For mobile app developers: Deploy faster vision-language capabilities on resource-limited devices by reducing visual token processing.
Authors
Xu Li, Yuxuan Liang, Yi Zheng, Zhe Liu, Xiaolei Chen, Haotian Chen, Rui Zhu, Fan Shi, Xiangyang Xue
Abstract
Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.