Medical image interpretation improved by smarter case selection in language models
Representation-guided in-context learning for medical image interpretation with multimodal large language models
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Understanding medical images is crucial but usually requires special training of AI models, which can be costly. The authors show a way to improve AI interpretation by choosing example cases that closely match the new images, without retraining the AI. Their method works well across different medical fields and tasks, making it easier to get accurate results quickly. It turns out picking the right examples matters more than picking many examples.
What this means in practice
- •For medical imaging software developers: Improve diagnostic accuracy by integrating alignment-based example selection into existing AI systems to interpret diverse medical images without costly retraining.
- •For autonomous vehicle sensor teams: Use representation-guided example retrieval in AI systems to interpret complex visual sensor data without retraining under new conditions or environments.
Authors
Minda Zhao, Fangyu Hu, Yan Luo, Yutong Yang, Jiahui Cai, Kaichen Zhou, Manling Li, Paul Liang, Yilun Du, Lucy Q. Shen, Mengyu Wang
Abstract
Medical image interpretation is central to diagnosis and care, yet adapting general-purpose multimodal large language models (MLLMs) often requires resource-intensive domain-specific fine-tuning. Here we introduce representation-guided in-context learning (RG-ICL), a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates. Across eight datasets spanning histopathology, radiology and retinal fundoscopy, RG-ICL improved classification (mean gain 20 percentage points) and visual question answering (VQA) (mean gain 13 percentage points) over no-context and conventional ICL, approaching or exceeding training-based comparators. Which cases were retrieved mattered more than how many: 6 query-aligned cases outperformed up to 32 randomly selected ones, whereas fixed or random cases often reduced accuracy below baseline. For VQA, aligning reference cases with both image content and question intent produced further gains. These findings indicate that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.