Summary
In colonoscopies, computers help find polyps by segmenting images, but they can make mistakes without any immediate warning. The authors propose a method called Referee-Based Quality Estimation (RBQE) that uses a second independently trained model to check how much it agrees with the main model on the same image. This agreement acts as a signal of reliability, helping identify uncertain or wrong segmentations in real time. They tested this method on a large collection of images and found that using different types of models for the referee works best to detect errors. This approach can help make automated polyp detection safer and more trustworthy during procedures.
What this means in practice
- •For medical device teams: Integrate RBQE to flag unreliable polyp segmentations in real-time colonoscopy systems, improving clinical decision support safety.
- •For machine learning engineers: Use cross-model agreement as a lightweight reliability metric for automated segmentation tasks without requiring ground-truth labels at inference.
Abstract
In real-time colonoscopy, ground-truth annotations are unavailable at inference, so polyp segmentation models can fail silently. We propose Referee-Based Quality Estimation (RBQE), a reference-free framework measuring agreement between a primary segmentation model and an independently trained referee on the same image. RBQE is evaluated on a standardized 1,223-image external benchmark drawn from four public datasets, using four referee configurations chosen to separate two design axes: referee independence and architectural diversity. Using a common Agreement Dice descriptor, a same-architecture referee differing from the primary model only in random initialization already yields a useful reliability signal (ROC-AUC = 0.923), showing that independent training alone is sufficient. Cross-architecture referees improve further: SegFormer-B0 achieves the strongest performance (ROC-AUC = 0.960), significantly outperforming the same-architecture control and UNet++, and exceeding a representative Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol, whereas a prompt-coupled MedSAM referee underperforms despite maximal architectural diversity. Because empty-mask agreement is trivially separable, we also report a restricted evaluation excluding such cases: ROC-AUC falls to 0.876 (SegFormer-B0, 1,046 images) and 0.783 (same-architecture control, 975 images), yet RBQE's margin over both baselines widens on this identical subset. RBQE additionally increases the mean Dice of retained predictions as low-agreement cases are progressively rejected, supporting selective prediction, and requires only one additional deterministic referee forward pass at inference. Our study therefore supports cross-model agreement as a practical, interpretable reliability framework for automated polyp segmentation.