Medical vision language models run efficiently on limited hardware
Density-Aware Calibration for Resource-Efficient Medical Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Medical vision-language models help computers understand clinical images and answer questions, but they usually need strong computers to work. The authors present CoreCalib, a way to choose small, smart sets of calibration data that help these models run well even when simplified to use less computer memory and power. CoreCalib picks diverse and representative examples from medical questions and images without needing extra training. Tests show CoreCalib improves model accuracy after compression compared to standard methods, making medical AI more practical for real-world clinics.
What this means in practice
- •For clinical ai engineers: Deploy medical vision-language models efficiently on hospital hardware with limited memory and compute by using CoreCalib for calibration data selection.
- •For medical image software developers: Integrate resource-efficient calibration for compressed medical vision-language models to maintain diagnostic question-answering accuracy without retraining.
Authors
Tien-Dat Dam, Huu-Thuan Huynh, Minh-Triet Tran, Trung-Nghia Le
Abstract
Medical vision-language models (VLMs) show promise for clinical image understanding, but their computational and memory requirements hinder deployment on resource-constrained clinical hardware. Post-training quantization (PTQ) offers a practical compression strategy without retraining, yet its effectiveness depends strongly on the calibration data. We propose CoreCalib, a training-free and quantizer-agnostic framework for constructing compact calibration sets that preserve the target medical VQA distribution. CoreCalib stratifies data by clinical question class, identifies representative visual modes using frozen DINOv2 features with UMAP and HDBSCAN, and expands selected images into class-balanced question-answer pairs. On Kvasir-VQA-x1, we evaluate CoreCalib with MASQuant, GPTQ, and QIG using a fixed 128-pair budget. CoreCalib achieves the best METEOR with all three quantizers and the best ROUGE-L with MASQuant and GPTQ. With W4A8 MASQuant, it improves METEOR from 0.5672 with COCO calibration to 0.6234, recovering approximately 41% of the degradation relative to BF16. These results support CoreCalib as a data-centric component for resource-efficient medical VLM deployment. Project page: https://magnusdtd.github.io/CoreCalib/ .