Brain MRI anomaly detection needs clearer evaluation steps

MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRI

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Detecting unusual regions in brain MRI scans without labels can be tricky because the way results are checked and measured varies and isn’t always clear. The authors created MIRTO, a method that carefully details and tests how each step in evaluating these detections affects the final scores. They showed that small changes in how maps are aligned or thresholds set can greatly impact the results. MIRTO tested many combinations and found which parts of the evaluation process matter most for fair comparison.

What this means in practice

  • For medical image analysts: Use MIRTO to get more reliable comparisons of anomaly detection methods by standardizing registration and threshold procedures in brain MRI evaluations.
  • For machine learning engineers: Apply MIRTO’s comprehensive evaluation pipelines to validate anomaly detection models more robustly and avoid misleading performance claims.

Authors

Negin Kafee Hernashki, Soumick Chatterjee

Abstract

Unsupervised anomaly detection (UAD) methods for brain MRI are ranked by a single score, yet that score rests on choices that are rarely reported: how each anomaly map is aligned with the reference, how and on which data the threshold is set, and which false-positive budget, metric, aggregation and lesion definition are used. We present MIRTO, an evaluation protocol that makes these choices explicit and measures their effect. It gates the geometry of every comparison with a registration check and label-free diagnostics of known power, sets thresholds on validation data alone and reports the false-positive volume actually realised on test, repeats each comparison over 15,552 defensible evaluation pipelines, and attaches paired subject-bootstrap intervals with multiplicity control. Applied to four UAD methods trained on the same healthy data and tested on 312 BraTS 2020 subjects, MIRTO showed that an axis-order mismatch between stored maps and the reference lowered a diffusion model's voxel AUROC from 0.873 to 0.583 whilst barely moving its slice-level AUROC. Within each metric, the method explained at least 0.95 of the variance in voxel AUROC and AUPRC and 0.77 in Dice, but only 0.14 in lesion sensitivity, where the lesion definition and hit criterion dominated. A Dice advantage that was significant at validation thresholds vanished at equal realised false-positive burden, and an exact identity attributes it to threshold transfer. A training-free change to REFLECT's latent aggregation raised Dice at equal burden by 0.052. Nine hypotheses were tested against explicit criteria; because the same cohort served to develop the protocol, all inference is exploratory.