When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?
Computer Vision and Pattern Recognition
Summary
The authors study models that use 3D CT scans and text prompts to identify multiple abnormalities without needing new labels, which helps when new data comes from different hospitals. They find that adapting these models at test time (TTA) only works well if the 3D structure the model expects is preserved and the model can already tell abnormalities apart. They introduce a new method called CARVE that helps the model figure out how many abnormalities are present and adapts efficiently using parts of the scan, improving performance on multiple CT tasks. Their work shows that adapting zero-shot 3D CT models to new data is a unique challenge and provides a new solution for it.
Authors
Ailar Mahdizadeh, Puria Azadi Moghadam, Xiangteng He, Leonid Sigal
Abstract
3D CT vision-language models (VLMs) classify abnormalities from text prompts in a zero-shot manner, enabling cross-institution deployment where labels are scarce and clinical tasks shift faster than supervised models can be retrained. A real CT scan, however, typically contains several co-occurring abnormalities, and the reliability of zero-shot multi-label prediction under distribution shift remains poorly understood. Test-time adaptation (TTA) updates a model on unlabeled target scans without source data or target annotations, yet existing TTA methods target multi-class softmax prediction on natural images or 2D medical segmentation, and none addresses unsupervised multi-label adaptation for zero-shot 3D CT VLMs. We study when TTA helps zero-shot 3D CT VLMs. A controlled diagnostic analysis shows that TTA is conditional: the volumetric input must preserve the encoder's depth structure, and the base representation must transfer to the target cohort, with depth reduction alone lowering internal AUROC by more than 0.12. We then focus on the regime where the base model already separates present from absent abnormalities. We introduce CARVE (Cardinality-Aware Retained-View Entropy), the first TTA method for this setting. CARVE estimates a sample-specific positive-label cardinality $\hat{k}$, optimizes a top-$\hat{k}$ objective to preserve co-occurring abnormalities, and performs memory-efficient multi-view adaptation by scoring weak 3D views without gradients before updating on a retained subset. Across contrastive CT-CLIP and anatomy-aware fVLM, CARVE provides the most consistent improvements across multi-label, three-class, and binary CT tasks when the base model is already discriminative. These results establish multi-label TTA for zero-shot 3D CT VLMs as a distinct problem and CARVE as a cardinality-aware solution.