Medical vision language models retain key image details after token pruning

MedSTP: Preserving Subspace and Transition Evidence for Medical VQA Token Pruning

Computer Vision and Pattern Recognition

Summary

Medical vision-language models answer questions about medical images using many small image pieces called tokens. The authors found a way to remove some tokens without losing important details like edges and textures that help answer questions accurately. They developed a method called MedSTP that keeps tokens representing the main visual features and those near boundaries. This improves efficiency while still performing well in tests on medical image question answering tasks.

What this means in practice

  • For medical imaging developers: Optimize vision-language models to handle medical images efficiently without losing critical visual evidence necessary for accurate question answering.
  • For radiology it teams: Deploy faster medical VQA systems that maintain accuracy by selectively pruning image tokens, reducing computation while preserving diagnostic detail.

Authors

Mengchen Fan, Qizhen Lan, Xi Xiao, Lin Zhao, Liang He, Tianyang Wang, Xiaoqian Jiang, Pulin Che, Baocheng Geng

Abstract

Medical vision-language models (VLMs) for visual question answering (VQA) often pass long visual-token sequences to the language model, making visual-token pruning attractive but risky for medical images, where answers may depend on localized boundary and texture evidence. We study training free pre-decoding pruning and treat pruning as a subspace stable evidence preservation problem. The retained tokens should remain close to the dominant visual evidence subspace induced by the full token set. To this end, we propose \emph{Subspace-Transition Token Pruning} (\method{}), which estimates each token's spectral support from principal component space and constructs a compact representation subspace. To better reflect medical image structure, \method{} further uses a bounded token-grid transition prior to promote boundary and texture sensitive tokens. We evaluate \method{} on the closed-ended SLAKE and OmniMedVQA benchmarks and on RadImageNet-VQA, which provides both closed-ended and open-ended evaluations, using Lingshu-7B and Qwen3-VL-8B backbones. \method{} delivers consistently strong performance on both the closed-ended and open-ended RadImageNet-VQA evaluations. These results show that \method{} is an effective training free pruning framework for medical domain VQA.