Visual token pruning efficiency depends on pruning method and context
Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count
Artificial Intelligence
Summary
Visual token pruning tries to make AI models that understand images work faster by removing less important parts of an image. The authors found that just measuring overall accuracy and counting tokens kept doesn’t show if pruning really picks the important parts or saves real computing time. They made new tests to check if pruning focuses on the right image parts and if it actually speeds up the model. Their experiments show that smaller token counts don’t always mean faster models, and different pruning methods perform differently depending on the situation.
What this means in practice
- •For ai developers: Choose pruning techniques that best balance speed and accuracy for specific visual language model applications to improve deployment efficiency.
- •For mobile app developers: Develop apps using visual models optimized with meaningful token pruning that ensures faster response times without losing important visual information.
Authors
Geng Li, Guohao Chen, Shilin Shan, Jianfei Yang
Abstract
Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM. Yet existing evaluations rely on aggregate task accuracy and retained token count as proxies for pruning quality and actual efficiency. We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens; many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded. 2) Retained token count is an unreliable proxy for efficiency, as identical token budgets can incur different end-to-end costs depending on where and how pruning is performed. Consequently, comparisons based on aggregate accuracy and retained token counts fail to recover the global performance-efficiency Pareto frontier across methods. We introduce PAA-Bench, a plug-and-play evaluation framework for existing benchmarks, to evaluate whether pruning succeeds through meaningful token selection and whether this success translates into real speedup. To determine when token choice actually matters, we propose Random-Pass@k to measure how often an example remains correct under random pruning, enabling comprehensive evaluation across different levels of selection difficulty. To compare efficiency fairly, we introduce Preservation-Constrained Speedup (PCS) to report the maximum end-to-end speedup each method achieves under a shared preservation target. Experiments across 8 methods, 3 VLM backbones, and 9 benchmarks show that stronger token compression does not necessarily yield greater speedup, method differences widen on examples where token selection matters, and efficiency rankings change with the preservation target.