Medical vision language models improve few shot uncertainty prediction
Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs
Computer Vision and Pattern Recognition
Summary
Making predictions with medical vision-language AI models can be tricky because it’s important to know how uncertain the model is about its answers. The authors point out that standard methods assume certain data conditions that don’t hold well when only a few examples are available to adapt the model to a new medical task. They propose a new method called AlignCP that adjusts how uncertainty is measured so the model stays reliable even when it learns from very few labeled examples. AlignCP works by aligning the model’s prediction confidence between the small labeled set and the unlabeled test set without needing extra labels.
What this means in practice
- •For medical imaging developers: Improve uncertainty estimates for AI models adapting to new medical imaging tasks using limited data.
- •For clinical ai deployment teams: Enhance reliability of medical AI systems in hospitals when transferring models to new diagnostic tasks with few labeled examples.
Authors
Xuan Cuong Ngo, Ngan Le
Abstract
Uncertainty estimation for medical vision--language models (VLMs) using conformal prediction has gained increasing attention due to its distribution-free coverage guarantees. However, standard conformal prediction relies on exchangeability between calibration and test data and typically requires a sufficiently large calibration set to obtain reliable coverage. These assumptions are difficult to satisfy in few-shot transfer settings, where only a small labeled support set is available to adapt a pretrained VLM to a new medical task, while an unlabeled query set is used for evaluation. Supervised fine-tuning on the support set changes the model parameters and consequently shifts the nonconformity score distribution, breaking exchangeability between calibration and query samples and leading to unreliable coverage under distribution shift. Existing transductive conformal adaptation methods often preserve validity by avoiding supervised updates. While this helps maintain conformal assumptions, it underutilizes the scarce labeled support data and limits task adaptation, which is the primary objective in few-shot learning. In this setting, conformal prediction should serve as an uncertainty estimation layer that supports the adapted model, rather than preventing adaptation itself. To this end, we propose AlignCP, a framework that reconciles supervised few-shot adaptation with conformal uncertainty estimation under non-exchangeability. AlignCP learns a reweighted calibration distribution that reduces the score-level discrepancy between the labeled support set and the unlabeled query set. By aligning the one-dimensional nonconformity score distributions, AlignCP aims to close the coverage gap induced by adaptation without requiring query labels.