Summary
In colonoscopy images, it is hard to know if computer programs that detect polyps are working well because there is no clear answer during use. The authors study a method called Referee-Based Quality Estimation (RBQE) that compares the results of two independently trained models on the same image to judge if the predictions are reliable. They find that even two models with the same design but different training can give a good signal of reliability, and models with different designs do even better. This approach can help doctors trust automated polyp detection by flagging uncertain cases without needing extra data. It only requires one more model check when the system is running, making it practical for real-time use.
polyp segmentationcolonoscopymachine learningmodel reliabilityreference-free evaluationcross-model agreementDice coefficientsegmentation modelreal-time inferenceselective prediction
Abstract
In real-time colonoscopy, ground-truth annotations are unavailable at inference, so polyp segmentation models can fail silently. We propose Referee-Based Quality Estimation (RBQE), a reference-free framework measuring agreement between a primary segmentation model and an independently trained referee on the same image. RBQE is evaluated on a standardized 1,223-image external benchmark drawn from four public datasets, using four referee configurations chosen to separate two design axes: referee independence and architectural diversity. Using a common Agreement Dice descriptor, a same-architecture referee differing from the primary model only in random initialization already yields a useful reliability signal (ROC-AUC = 0.923), showing that independent training alone is sufficient. Cross-architecture referees improve further: SegFormer-B0 achieves the strongest performance (ROC-AUC = 0.960), significantly outperforming the same-architecture control and UNet++, and exceeding a representative Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol, whereas a prompt-coupled MedSAM referee underperforms despite maximal architectural diversity. Because empty-mask agreement is trivially separable, we also report a restricted evaluation excluding such cases: ROC-AUC falls to 0.876 (SegFormer-B0, 1,046 images) and 0.783 (same-architecture control, 975 images), yet RBQE's margin over both baselines widens on this identical subset. RBQE additionally increases the mean Dice of retained predictions as low-agreement cases are progressively rejected, supporting selective prediction, and requires only one additional deterministic referee forward pass at inference. Our study therefore supports cross-model agreement as a practical, interpretable reliability framework for automated polyp segmentation.