AI summaryⓘ
The authors studied why certain AI models that use natural language prompts to segment medical images don't work well in clinical settings. They believe the problem isn't just a lack of medical data or how prompts are worded, but a deeper issue with using natural language for guidance when there isn’t much paired image-text data available. To fix this, they developed a method called Few-Shot Concept Prompt Learning (FS-CPL), which learns visual prompts directly from a few example image-mask pairs without retraining the whole model. Their approach improved segmentation accuracy across different medical imaging tasks and worked with different versions of the model, showing it's a helpful addition to existing methods.
Promptable segmentationFoundation modelsNatural language promptsFew-shot learningVisual concept promptsMedical imagingDice scoreMask supervisionEncoder-decoder backboneDomain-specific pretraining
Abstract
Promptable segmentation foundation models (FMs) such as SAM3 and Medical SAM3 promise few-shot, interactively-specified segmentation for medical imaging through a natural language interface, yet their performance on clinical tasks falls well short of this promise. We posit that this shortfall is not an artefact of insufficient medical pretraining or imperfect prompt phrasing, but a structural limitation that will persist in any domain where paired image-text supervision is scarce, as it is across most clinical modalities. We further hypothesize that the limitation is specific to natural language as a control signal: a visually grounded prompt, learned directly from the target distribution, should recover the lost performance without additional image-text data or backbone retraining. We propose Few-Shot Concept Prompt Learning (FS-CPL), which learns a continuous concept prompt embedding $\mathbf{p}^* \in \mathbb{R}^{T \times d}$ from a small support set of $K$ image--mask pairs via mask supervision, with the encoder-decoder backbone frozen. Across four public benchmarks spanning ultrasound and endoscopy (BUSI, HC18, TN3K, CVC-Clinic), FS-CPL delivers absolute Dice improvements of up to $+0.62$ over canonical text prompts and is \emph{backbone-agnostic}: it lifts both vanilla SAM3 and the domain-specifically pretrained Medical SAM3, showing that visual concept prompting is complementary to in-domain pretraining.