Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of adapting vision-language models (VLMs) to new medical images at inference time without labeled data, which is difficult due to noisy updates harming pretrained features. They propose a method called MSSA that does not change the model's parameters but instead builds a memory of reliable image-text predictions and uses these as guides to better segment medical images. MSSA also aligns new images with structurally similar memory examples to improve results. Their experiments show that MSSA improves segmentation accuracy more consistently and effectively than existing methods that fine-tune models during testing.
test-time adaptationvision-language modelsmedical image segmentationdistribution shiftsemantic priorscross-image alignmentprototype alignmentnoise filteringunsupervised learningpretrained features
Authors
Lingrui Li, Nan Pu, Dong Zhao, Wenjing Li, Andrew P French, Zhun Zhong, Xin Chen
Abstract
Test-time adaptation (TTA) aims to mitigate distribution shifts by adapting models with unlabeled target data at inference time. While TTA with vision-language models (VLMs) has shown promising results in classification, extending it to medical image segmentation remains challenging. In this setting, the adaptation gains from optimizing on VLM-generated predictions are often outweighed by the degradation to the VLM's strong pretrained features caused by noisy, update-driven learning, resulting in limited and unstable improvements. We therefore propose Memory-Supported Synergistic Adaptation (MSSA), a novel training-free TTA framework for medical image segmentation. Without updating model parameters, MSSA dynamically selects reliable image-text predictions to construct an online memory, uses them as text-guided semantic priors, and couples them with cross-image structural alignment for robust adaptation. Specifically, MSSA consists of (i) a noise-aware memory construction module that filters and stabilizes cross-modal predictions, and (ii) a relevance-driven prototype alignment module that aligns the target sample with structurally consistent memory samples and their reliable predictions to improve adaptation. Extensive experiments on multiple medical segmentation benchmarks demonstrate that MSSA consistently improves VLM-based segmentation models and outperforms existing fine-tuning-based TTA methods by a clear margin, with gains of up to 12.2% DSC and 11.7% mIoU. Project page: https://lingrayy.github.io/MSSA/ .