Cancer segmentation AI signals errors through internal patterns
Medical AI Encodes a "Feeling of Error": Verifying Cancer Segmentation via Internal Concepts
Computer Vision and Pattern Recognition
Summary
Cancer detection AI can sometimes make mistakes that look convincing but are wrong, which can cause harm. The authors studied whether these AI systems have an internal 'feeling' that tells them when they might be wrong. They found that by looking inside the AI’s inner workings and spotting certain patterns, they could tell when the AI was likely to fail. This method works better than just looking at the AI’s confidence scores and also explains why the mistake happened. They tested this on models for prostate, pancreatic, and brain cancer images.
What this means in practice
- •For medical imaging teams: Identify when cancer segmentation AI models are likely making errors using their internal signals to avoid wrong diagnostics.
- •For ai safety engineers: Develop more reliable AI by monitoring internal concept patterns to predict and explain model failures during medical image analysis.
Authors
Mengmeng Ma, Yunxiang Peng, Tang Li, Lu Lin, Binsheng Zhao, Oguz Akin, Xi Peng
Abstract
Cancer segmentation models can fail silently, generating plausible but incorrect masks that risk missed findings or unnecessary biopsies. A critical question arises: Do AI models "know" when they are wrong, and if so, can we use the signal to predict their own failures? Humans do have a "Feeling of Error" (FOE): a spontaneous sense of unease that flags a potential error during thinking. We investigate whether cancer segmentation models exhibit an analogous internal signal. Unlike output-level cues (e.g., prediction confidence or uncertainty), which offer no insight into why a failure occurs and suffer from a sensitivity-quality tradeoff where high detection sensitivity could degrade overall segmentation quality. We instead propose to capture the model's FOE from its inner workings. Using mechanistic interpretability tools, specifically Sparse Autoencoders, we decompose internal neural activations into a dictionary of human-interpretable concepts and show that failure cases exhibit a distinct latent signature: fewer active concepts with lower activation magnitudes compared to successful segmentation. By training a classifier on these concept activations, we achieve accurate failure detection along with explanations for the model's mistakes. Experiments on prostate, pancreatic, and brain cancer segmentation demonstrate that our approach outperforms output-based methods in failure detection while preserving segmentation quality.