PhonoBridge improves silent speech decoding with phoneme based approach
PhonoBridge: Error-Aware Phoneme-Mediated EMG-to-Text Reconstruction
SoundComputation and LanguageHuman-Computer Interaction
Summary
Decoding silent speech signals collected from muscles (EMG) into text is hard because there isn’t much paired data to learn from. The authors propose PhonoBridge, which first converts muscle signals into sounds (phonemes) and then uses a language model to turn those sounds into text. They also teach the language model to handle errors by simulating common mistakes in phoneme sequences. This approach reduces errors in silent speech decoding compared to previous methods.
What this means in practice
- •For assistive technology developers: Develop devices that convert silent muscle signals into text for communication aids with improved accuracy.$Commercial implications: Enables commercial silent speech interfaces for people with speech impairments by improving decoding accuracy and readable output.
- •For human computer interface engineers: Integrate phoneme-aware EMG decoding with language models to improve silent speech input methods for hands-free control.
Authors
Wen Qiu, Jiacheng Zhang, Xuanda Chen, Siqi Cai, Haizhou Li
Abstract
Electromyography (EMG) provides a non-invasive means of decoding silently articulated speech into text, but accurate decoding remains challenging due to limited availability of paired EMG-text data. We introduce PhonoBridge, a two-stage framework comprising a channel-aware EMG-to-phoneme frontend and a large language model (LLM) backend for text reconstruction. The frontend is trained on paired EMG-text data that consists of channel-wise feature extraction, cross-channel fusion, and electrode embeddings. With phonemes as the intermediate representation, we expand the backend's training data by converting LibriSpeech transcripts into phoneme-text pairs without additional EMG recordings. To adapt the LLM to frontend noise, we introduce Error-Profile-Guided Augmentation (EPGA), which perturbs transcript-derived phoneme sequences according to error patterns estimated from frontend predictions, generating additional training pairs for the LLM. On the Open Vocabulary silent-speech dataset, PhonoBridge achieves 24.77% WER, outperforming the previously reported best of 28.27% on the same test split.