Brain MRI model features fail across datasets due to content encoding
Dataset-Specific Encoding of Image Content Underlies Cross-Dataset Failure in Brain MRI Foundation Models
Computer Vision and Pattern Recognition
Summary
Brain MRI models that are trained and then fixed often fail to work well on new unseen data. The authors found that this failure is not because the model learns which dataset a scan comes from, but because the actual brain images are represented differently in each dataset. Most models encode the brain image content in dataset-specific ways, causing problems when applied to new data. Only a few broadly trained models can encode images consistently across datasets.
What this means in practice
- •For medical imaging engineers: Improve MRI model design by testing if features encode brain content consistently across datasets to enhance cross-dataset reliability.
- •For machine learning platform teams: Use broadly pretrained models like 3DINO or SAM-Med3D to build MRI analysis systems that better handle new datasets without retraining.
Authors
Minh Sao Khue Luu, Bair N. Tuchinov
Abstract
Frozen features of brain MRI foundation models lose accuracy on new datasets, a failure usually blamed on the features encoding where each scan came from. We show that the failure lies elsewhere. Across 17 public datasets and eight frozen representations, every representation recognizes the MRI sequence within each dataset, yet most fail on an unseen one. The cause is neither dataset encoding nor a shift of each dataset: the content itself is encoded along different directions in each dataset. This explains which datasets fail, for the sequence and for sex, and why removing the dataset signal with INLP, LEACE, CORAL or adversarial training never improves sequence transfer. Only the broadly pretrained 3DINO and SAM-Med3D encode the sequence consistently enough to transfer; the brain-specific BrainIAC and BrainFM do no better than an untrained network. Code is available at https://github.com/BrainFM/brainfm-profiler/tree/main/representation_utility.