CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift
2026-06-22 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors address the challenge of safely updating medical vision-language models when adding new imaging types, which can lead to losing ability on old data or relying on shortcuts. They propose CADRE, a method that adapts models efficiently without forgetting previous knowledge or drifting too far from the original reliable model. Tested on breast cancer images from very different types of scans, CADRE improved accuracy and reduced forgetting much more than other methods. They focus on stability traits important for clinical safety but do not claim it handles all real-world issues like unexpected data or attacks.
vision-language modelscatastrophic forgettinglow-rank adaptationelastic weight consolidationcross-modalityembedding driftbreast cancer imagingbackward transfermodel adaptation
Authors
Amrita Singh, Rishabh Jha
Abstract
Medical vision-language models (VLMs) such as BiomedCLIP generalize broadly, but adapting them to a clinical service is as much a safety problem as an accuracy one. Updating a deployed model for a new imaging modality can fail silently in two ways that harm patients: it can forget modalities it already handled (catastrophic forgetting), and it can drift from its trustworthy pretrained prior toward modality-specific shortcuts. We study parameter-efficient continual adaptation through these two properties rather than leaderboard accuracy, presenting CADRE: a frozen-backbone framework combining low-rank adaptation (LoRA) with an online, self-scaling, similarity-aware elastic weight consolidation term that bounds retained-competence loss, and an anchor-to-prior penalty bounding embedding drift from the frozen prior. Two short guarantees, a bound on total consolidation mass and a scale-invariance property, remove the scale-related sources of vanilla EWC's order fragility. Using breast cancer across three maximally dissimilar modalities (histopathology, ultrasound, chest radiography) as a controlled cross-modality stress test, under a multi-seed, multi-order protocol with paired significance testing and training approximately 0.23% of parameters, CADRE attains the highest accuracy, SPQ, and backward transfer and the lowest forgetting among adapting methods, reducing forgetting roughly sevenfold versus the strongest regularized baseline (0.075 to 0.011; paired p=0.023) and achieving positive backward transfer where every baseline is negative. We frame these as stability properties aligned with clinical-safety desiderata, not a deployment guarantee; robustness to distribution shift and adversarial inputs is out of scope.