Speech brain computer interfaces work well without large models
Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling
Computation and Language
Summary
Speech brain-computer interfaces (BCIs) let people control devices by decoding their brain signals related to speech. The paper finds that huge, complex models are not always needed to accurately translate brain signals into text. The authors created a smaller, smarter system called BEST that uses intermediate steps to better understand signals and directly produce text, working well even with limited data and computing power. Their approach reduces errors and runs more efficiently than bigger models, showing that careful design can beat simply making models larger.
What this means in practice
- •For speech technology developers: Create efficient neural decoding software that converts brain signals to text with lower computation and data needs.
- •For mobile device developers: Implement low-resource brain-to-text systems for wearable or portable devices where computing power is limited.
Authors
Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, Ting Han
Abstract
Recent speech brain-computer interfaces (BCIs) increasingly rely on large models and complex multi-stage pipelines for neural decoding. However, in speech BCI, where training data are scarce, noisy, and expensive to collect, the benefits of model scaling remain unclear. This work revisits a fundamental question: are large models truly necessary for accurate neural decoding? We propose BEST (Brain-Encoding-Speech-Text), a two-stage framework for efficient neural-to-text decoding. Instead of relying on large generative models or phoneme-centric pipelines, BEST introduces hierarchical intermediate representations and a compact speech-oriented decoder that directly maps neural activity to text. This design preserves essential structure while significantly reducing computational cost, enabling robust generalization in low-resource settings and practical deployment under constrained resources. On the Brain-to-Text '24 and '25 benchmarks, BEST achieves word error rates (WERs) of 6.19% and 1.62%, respectively, using only 3.85% of the parameters required by comparable methods. Beyond these results, we provide three key insights: (1) decoder scaling yields non-monotonic gains; (2) cross-modal alignment is more critical than model size; and (3) overfitting degards sequence-level performance. Representation analysis further reveals a consistent neural-audio-text alignment pathway.