Foundation models adapt better than task models for new medical image sites
Medical Image Segmentation: Foundation Model Adaptation versus Task-Specific Training
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical image segmentation helps doctors find important areas in pictures like scans or photos of the body. The paper compares two ways to train computers for this: adjusting big pre-trained models called foundation models, or training special models just for each task. The authors found that foundation models work better when tested on images from different hospitals, though task-specific models sometimes find smaller details better inside familiar data. Also, only a small part of the foundation model needs to be changed depending on the type of medical images.
What this means in practice
- •For hospital imaging teams: Deploy foundation models adapted for different sites to improve segmentation accuracy on varied medical image data.
- •For medical ai developers: Optimize foundation model adaptation for specific imaging domains to balance training cost and segmentation performance.
Authors
Emre Kutlusan, Yongmin Li
Abstract
Foundation models pretrained on large natural-image collections are increasingly adapted for medical image segmentation, in the expectation that they will replace models trained for each task. Most comparisons, however, use data from the training distribution and a single task-specific baseline whose configuration is rarely justified, and therefore provide limited evidence of performance at new clinical sites. It also remains unclear how much of a foundation model must be adapted, and what determines this. We adapted a foundation model, DINOv2, with strategies ranging from tuning only its bias terms to full fine-tuning, and compared it with U-Nets and ResU-Nets trained from scratch. Experiments covered colonoscopy, breast ultrasound and dermoscopy, with testing on held-out data from the training source and on independent datasets from other institutions. The adapted foundation model generalised better on every external dataset. Within the training distribution, the task-specific models remained competitive when training data were abundant, and delineated boundaries more precisely, while the foundation model more often missed small lesions. Only a small fraction of the foundation model required adaptation, and the capacity needed was determined by the imaging domain rather than by the adaptation method. A widely used polyp test set was also found to duplicate images from another benchmark. Foundation-model adaptation is therefore warranted where models must work across sites, with capacity selected on the target domain.