Retrieval augmented method improves medical image segmentation corrections
Retrieval-Augmented Correction for Generalist Medical Image Segmentation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical imaging often uses computer programs to outline important body parts automatically, but these outlines sometimes need fixing by humans. The authors introduce GRACE, a tool that improves these outlines by looking at a database of verified past cases and using similar shapes to guide corrections. This tool can fix multiple organs with one model instead of needing separate fixers for each type of anatomy. It works best when the initial outline is less accurate and shows promise across different imaging types, although challenges remain in fully adapting to new imaging methods. This approach could simplify improving medical image analysis.
What this means in practice
- •For radiology software developers: Enhance segmentation results across multiple organs using a single correction tool guided by a database of verified cases.
- •For medical imaging service providers: Offer improved automated correction services for clinical imaging workflows without needing specialized correction models per anatomy.$Commercial implications: Enables scalable correction tools for various anatomical structures that can be marketed to hospitals and imaging centers.
Authors
Vincenzo Marcianò, Xiaoming Zhang, Michela Antonelli, Sebastien Ourselin, Maria A Zuluaga
Abstract
Automated segmentation is routinely used as the first step in quantitative image-based clinical workflows, where predicted masks are subsequently reviewed and corrected by human operators. Automated correction of the residual structural errors is currently addressed one anatomy at a time, so the number of trained correctors scales with the number of segmentation targets. We present GRACE, a retrieval-augmented refiner that edits an input mask under context drawn from a shared knowledge base of verified prior cases. The refiner is trained once on a multi-organ corpus spanning compact organs, cavities, branching vessels, and tumour lesions. In the held-out dataset-transfer evaluation, three held-out datasets query a fixed knowledge base built from training data only. A separate leave-one-anatomy-out experiment tests parameter-free index extension: verified cases of an unseen solid organ enter the knowledge base, and the refiner stays frozen. A light shape-aware projection head on a frozen multimodal encoder maps embedding similarity to the shape agreement of the masks, so the knowledge base returns cases that the refiner can use as a shape prior. We evaluate GRACE on 18 public datasets spanning CT, MRI and ultrasound, using nine refinement configurations and seven upstream segmentors that span weak, mid, and strong operating regimes. We report each change with the DSC of its input mask. On the weak tier, the input DSC is 0.386 and GRACE adds up to +0.35 DSC. On a strong segmentor, the input DSC is 0.932 and the mean change is small. Transfer to an imaging modality that is absent from training and from the index is partial and depends on the anatomy, and remains an open research direction. Our results suggest retrieval-conditioned refinement is a practical alternative to anatomy-specific correction models. Code: https://github.com/robustml-eurecom/grace