Stress-Testing EEG Foundation Models for Clinical Decoding: Dataset Identity and Targeted Negative Controls
2026-07-27 • Machine Learning
Machine LearningArtificial IntelligenceNeural and Evolutionary Computing
AI summaryⓘ
The authors tested six pretrained EEG models to see how well they perform on different clinical tasks and datasets. They found that results vary a lot depending on how the models are evaluated, which data is used, and the comparison methods chosen. For example, one model called REVE did not always outperform simple or random methods, showing that pretrained models might not always be better for certain tasks like diagnosing dementia or Alzheimer's disease. Only in one clear case—detecting seizures across subjects—did REVE show a definite advantage. Overall, the authors highlight the importance of careful testing and controls when assessing EEG models.
EEGpretrained modelAUROCdataset shifttransfer learningictal detectionrandom featurescross-subject evaluationlinear probefoundation model
Authors
Marzieh Zare
Abstract
Pretrained EEG foundation models are increasingly proposed for clinical decoding, but their transfer across populations and robustness to negative controls remain unclear. We benchmark six models (LaBraM, EEGMamba, CBraMod, REVE, BENDR, and BIOT) on five clinical tasks across four datasets using frozen linear probes with leave-one-subject-out, subject-grouped, or explicitly identified recording-level splits. Selected REVE findings are tested against random initialisation, random features, label permutation, scrambled-label fine-tuning, and projection sensitivity. On Korean dementia (CAUEEG, three-way), frozen REVE reaches 0.568 AUROC versus 0.769 for classical features; the ordering persists on a patient-disjoint held-out split (0.565 versus 0.768). Dataset identity is readily decoded from frozen embeddings (AUROC 1.000 at PCA-50; 0.9998 after band restriction and per-epoch z-scoring), whereas the same PCA-50 pipeline decodes Korean diagnosis at 0.528. A randomly initialised encoder also outperforms pretrained REVE on this task (0.659 versus 0.570). On Alzheimer's disease, Gaussian random projection and PCA of the same pretrained embeddings perform similarly, and classical features nominally exceed REVE at the subject level. The clearest controlled positive is cross-subject ictal detection on CHB-MIT (n=23), where REVE achieves 0.793 AUROC, 9.2 percentage points above a randomly initialised encoder. These results show that EEG foundation-model conclusions depend strongly on evaluation unit, dataset shift, comparator strength, and targeted controls.