EEG model controls reveal limits of pretrained and random features
Controlled Evaluation of EEG Foundation Models: Random Initialisation, Band-Power and Validation-Selected Probes
Machine LearningArtificial Intelligence
Summary
This paper looks at how well certain brainwave (EEG) models perform when tested carefully with proper controls. The authors find that sometimes random versions of these models can do just as well as trained ones, especially on tasks like sleep and emotion recognition. They also show that simple hand-crafted features can outperform these complex models in some cases. Their findings suggest we need better ways to check these models before trusting their results.
What this means in practice
- •For biomedical engineers: Assess EEG model performance using strong controls to avoid misleading improvements on sleep and emotion detection tasks.
- •For machine learning engineers: Evaluate EEG model pretraining benefits with protocols that compare against random initializations and conventional spectral features.
Authors
Mingyu Jeon, Suwan Cho, Jae Young Suh
Abstract
EEG foundation models report downstream gains, yet their protocols seldom include three controls. These are the same encoder untrained, spectral features fitted on the same data, and a layer picked without looking at test scores. We therefore evaluate five publicly released EEG foundation models and an in-house JEPA-style model under one protocol with all three controls and several seeds, on five benchmarks and four added task families. The controls change the reading in three ways. First, picking the probed layer by its test score raises the layer-wise peak by an average of +0.034 and reverses the peak-versus-output gap in 9 of 36 cells. Second, with the layer picked on validation, random initialisation matches or beats pretraining for half of the models on sleep and emotion, and no model beats a five-band log-power baseline on emotion or motor imagery. Third, after full fine-tuning only 5 of 30 pairs gain beyond twice their seed spread, and the one clear loss of -0.290 AUROC persists under matched learning rates. From these results we draw four reporting practices, and we release every cell for re-analysis.