Phoneme-guided method improves speech recognition for low-resource languages
Phoneme-Guided Initialization for LLM-based Speech Recognition
Computation and LanguageSound
Summary
Speech recognition systems usually work best when they have a lot of spoken and written examples to learn from. The authors found that breaking down the process into identifying sounds first and then converting those sounds to written words works better when there's not much data available. They created a new approach that trains one part of the system on sounds and another on word spelling before combining them to improve accuracy. Tests on languages like Japanese, Chinese, Tatar, and Urdu showed this method is as good as or better than older methods.
What this means in practice
- •For speech technology teams: Improve automatic speech recognition accuracy for languages with limited paired speech-text data by using phoneme-guided pre-training methods.
- •For language technology developers: Develop speech recognition tools for low-resource languages by incorporating phoneme-based initialization to boost performance with scarce data.
Authors
Ryo Magoshi, Shinsuke Sakai, Tatsuya Kawahara
Abstract
Speech large language models (speech LLMs) perform well on automatic speech recognition (ASR) when sufficient paired speech-text data is available, but their performance degrades in low-resource settings. A cascaded pipeline that performs speech-to-phoneme (S2P) conversion followed by phoneme-to-grapheme (P2G) conversion has been shown to outperform end-to-end speech LLMs in this regime, suggesting that phoneme-mediated processing is beneficial when paired data is scarce. We propose \textit{phoneme-guided initialization}, a simple method that uses this insight within an end-to-end framework: we pre-train the audio encoder on S2P and the LLM on P2G tasks, then connect them and fine-tune the full model end-to-end on the target ASR task. Experiments on Japanese (CSJ), Chinese (AISHELL-1), and two low-resource languages from Common Voice 25.0 (Tatar and Urdu) show that our method matches or outperforms both the cascaded S2P-P2G baseline and the end-to-end model without P2G initialization.