Visual token pruning improves efficiency of multi image vision language models

RI-Prune: Relation and Information Aware Visual Token Pruning for Multi-Image Vision-Language Models

Computer Vision and Pattern Recognition

Summary

Handling multiple images at once in large vision-language models creates too many visual tokens, which slows down processing. Existing methods to reduce tokens rely on attention signals that work well for single images but fail for multiple images, causing performance drops. The authors tested and found attention is a weak way to pick important tokens for multi-image tasks. They propose RI-Prune, a new method that scores tokens using multiple signals related to their importance and relations across images, without needing training or attention. This method keeps more useful information while reducing tokens, improving speed and accuracy on multi-image benchmarks.

What this means in practice

  • For ai model engineers: Optimize multi-image vision-language models to run faster while maintaining accuracy by improving token pruning without retraining or complex attention.
  • For multimedia content platforms: Enhance processing speed and resource efficiency for systems analyzing multiple images simultaneously, such as content moderation or tagging services.

Authors

Dongjin Lee, Eunju Lee, Jinhee Jang, Yoonji Lee, Youngbin Kim

Abstract

Large vision-language models (LVLMs) have recently been extended to handle multiple input images, but this extension dramatically increases the visual token budget, making visual-token pruning not merely an efficiency option but a practical necessity. Existing visual-token pruning methods, primarily designed for the single-image setting, rely on attention-derived saliency signals to estimate per-token importance. However, this attention-centric design transfers poorly to the multi-image regime: it causes substantial performance degradation on multi-image benchmarks, suffers from strong positional bias, and loses discriminative attention signals as the number of input images increases. To test whether attention-selected tokens are truly important for multi-image reasoning, we conduct two complementary causal probes Leave-One-In and Leave-One-Deletion, which measure token importance by restoring individual tokens to a zeroed visual sequence or removing them from the full sequence. Attention-based pruners provide only marginal gains over random selection, showing that attention is a weak proxy for visual-token importance in multi-image reasoning. Motivated by this gap, we propose Relation and Information Aware Pruning (RI-Prune), a training-free, attention-free pre-decoder selector for multi-image visual token pruning. RI-Prune scores each visual token using a z-normalized combination of three signals computed from projector outputs: query relevance, cross-image correspondence with compact per-image prototypes, and intra image representativeness. By integrating these signals, RI-Prune performs balanced per-image Top-K selection to preserve information across input images. Across multiple multi-image benchmarks and three LVLM backbones, RI-Prune achieves a stronger trade-off between token reduction and task performance than prior pruning methods, particularly under aggressive token budgets.