Multi-subject pretraining improves personalized silent speech decoding accuracy
Multi-Subject Pretraining Enables Short-Calibration Personalization for Closed-Corpus Surface EMG Speech Decoding
Machine Learning
Summary
Listening to muscle activity signals to understand silent speech faces challenges because everyone's signals are different. The authors show that training a computer model on data from many people first, then fine-tuning it with just a few minutes of data from a new person, greatly improves accuracy. This approach works well even with very little personalized data and simplifies the calibration process. The study focused on a fixed set of sentences and standard sensor placement to keep the task controlled.
What this means in practice
- •For assistive technology developers: Create silent speech devices that require only a few minutes of user calibration for personalized, accurate decoding in fixed phrase contexts.$Commercial implications: Enables marketable silent speech communication aids with short setup times for users with speech impairments.
- •For wearable device engineers: Design muscle-sensing wearables using multi-subject pretrained models to quickly adapt decoding for new users with minimal data.
Authors
Chenqian Le, Beatrice Fumagalli, Yasamin Esmaeili, Xupeng Chen, Tianyu He, Nikasadat Emami, Adeen Flinker, Yao Wang
Abstract
Surface electromyography (sEMG)-based silent speech interfaces are limited by cross-user variability and calibration burden. We study a limited-data setting in which each of 27 speech-typical participants contributed less than 0.5 h of data (21.3 min on average) across Aloud and Mimed speech. Within a closed 50-sentence corpus, we used leave-one-subject-out evaluation, initializing from a released single-subject checkpoint, pretraining on non-held-out participants, and fine-tuning on the target participant. This pipeline achieved 21.7% character error rate (CER) and 31.9% word error rate (WER), compared with 49.3% CER without target-subject calibration and 68.0% CER for direct checkpoint fine-tuning. Multi-subject pretraining from random initialization followed by fine-tuning reached 44.9% CER and did not converge under the fixed schedule in 5 of 27 folds, indicating substantial optimization and accuracy benefits from checkpoint initialization. Macro-averaged CER declined from 74.4% with one pretraining participant to 21.7% with 26. Three minutes of target-subject calibration achieved 20.5% CER and 31.7% WER, with no statistically significant difference from the full approximately 13-min pool (21.7% CER and 31.9% WER). A subject-specific adapter provided no detectable benefit. Excluding the five evaluation sentences from all sEMG model-training data increased CER and WER to 78.6% and 99.9%. These results support short-calibration personalization in a standardized-montage, closed-corpus setting.