Visual token pruning reduces vision language model costs efficiently
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Large vision-language AI models process many parts of an image called visual tokens, which takes a lot of computing power. The authors found that cutting down these tokens too much can hurt performance because of biases and data loss. They created VPRune, a method that carefully picks which tokens to keep without needing extra training. This keeps the model accurate while making it run faster, especially on devices with limited resources.
What this means in practice
- •For mobile app developers: Use VPRune to reduce the runtime and energy costs of vision-language models in mobile apps without losing accuracy.
- •For product teams for smart cameras: Incorporate VPRune to speed up image understanding on smart cameras with limited computing power while keeping high accuracy.
Authors
Guangchuan Lv, Dianxing Shi, Dingjie FU
Abstract
Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose \textbf{VPRune}, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.