EEG foundation models show varied motor imagery results across setups
Matched-Input Estimates Differ in Sign Across Architectures: Auditing EEG Foundation Models on Motor Imagery
Machine Learning
Summary
The paper studies how brain signal models that are trained in advance perform on tasks where people imagine moving parts of their body. The authors found that traditional models trained specifically for this task often do better than general pre-trained models. They also discovered that differences in how the brain signals are prepared before inputting to the models can change results differently depending on the model. This suggests that no single testing method can fairly compare these models. Finally, adjusting the models for more confidence calibration can improve reliability even if accuracy is lower.
What this means in practice
- •For brain-computer interface developers: Use matched-input evaluation to select EEG models better suited for motor imagery tasks in BCI systems.
- •For medical device engineers: Improve EEG signal preprocessing protocols to optimize model performance depending on architecture in motor imagery applications.
Authors
Kevin Zhou, Sparsh Roy
Abstract
Pretrained EEG foundation models are increasingly proposed as general-purpose encoders for brain-computer interfaces, yet recent benchmarks disagree about when their representations transfer to downstream tasks. We audit LaBraM and CBraMod on motor imagery under a validation-locked protocol in which preprocessing, architecture, optimization, freeze depth, checkpoint, temperature, and method selection are determined using training-session data only. On four-class BCI Competition IV-2a, every supervised comparator evaluated here outperforms every foundation-model configuration, including validation-selected fine-tuning. We then examine a key confound: foundation models and task-specific decoders are normally evaluated using different input pipelines. Retraining three supervised architectures on the broadband arrays consumed by the foundation models produces matched-input accuracy differences of opposite sign across architectures: broadband input improves ATCNet by 0.078 accuracy while reducing EEG Conformer accuracy by 0.088. None of the three individual matched-input terms is significant after multiple-comparison correction at n = 9, so we treat the sign variation descriptively rather than as a formal architecture-by-pipeline interaction. These observed sign differences suggest that a single comparator may not provide an architecture-invariant decomposition of a pretrained-versus-supervised performance gap. The four-class deficit also does not reproduce uniformly across motor-imagery datasets: on two-class BNCI2014-004 we cannot detect the same separation between fine-tuned CBraMod and the supervised comparators. Finally, validation-fitted temperature scaling returns foundation-model calibration error to the supervised range despite substantially lower four-class accuracy.