Multi-teacher training improves real-time speech recognition for electrolarynx users
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
Sound
Summary
Speech recognition systems usually struggle to understand electrolaryngeal speech, which is speech produced using a medical device after someone loses their voice box. The authors designed a new way to train a small, fast speech recognition model by learning from two larger models: one trained on normal speech and another fine-tuned on electrolaryngeal speech. This method helps their model better understand both healthy and electrolaryngeal speech in real-time on simple devices. Their model cuts errors in electrolaryngeal speech recognition from about 39% down to 21%.
What this means in practice
- •For mobile device developers: Develop on-device speech recognition apps that work well for electrolaryngeal speech without heavy computing resources.$Commercial implications: Enables real-time speech recognition apps for electrolaryngeal users running efficiently on consumer smartphones.
- •For assistive technology designers: Create portable communication aids that recognize electrolaryngeal speech accurately and respond instantly.
Authors
Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf, Philipp Aichinger, Martin Hagmüller
Abstract
Self-supervised learning (SSL) has improved speech representations, yet performance degrades in pathological domains such as electrolaryngeal (EL) speech, and the computational footprint of SSL models limits their applicability in real-time, on-device deployment. We propose a multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech. Two teachers are distilled progressively: a frozen SSL model providing discrete phonetic cluster targets from HE speech, and an EL-fine-tuned speech recognition model supplying continuous bottleneck feature targets. Evaluated via downstream speech recognition, our approach reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline. Among causal convolutional, Transformer, Conformer, and Mamba-based student architectures, a Mel-Conformer achieves the best combination of EL accuracy and computational efficiency. The final encoder contains 21.9,M parameters and runs at a real-time factor of 0.30 under ONNX Runtime on a single CPU core.