T-SANDHI improves speech recognition for Taiwanese Hokkien tones
T-SANDHI: Tone Sandhi-aware Adaptive Network with Decoupled Hybrid Injection for Low-resource Taiwanese Hokkien Speech Recognition
Computation and Language
Summary
Tone changes in Taiwanese Hokkien spoken language can confuse speech recognition systems. The authors found that existing models actually understand these tone changes well, but mix them up with similar-sounding tones. To fix this, they designed a method called T-SANDHI that separates the sound variations from the intended words, using a special model that combines two ways of hearing tones. This approach helps the system recognize Taiwanese Hokkien speech more accurately, even with limited data.
What this means in practice
- •For speech recognition developers: Create more accurate automatic speech recognition systems for Taiwanese Hokkien by separating tone variation from intended words.
- •For low-resource language technologists: Improve speech recognition in languages with complex tone changes by applying adaptive models that handle tonal confusion explicitly.
Authors
Hung-Yang Sung, Chien-Chun Wang, Tien-Hong Lo, Yu-Sheng Tsao, Yung-Chang Hsu, Berlin Chen
Abstract
In Taiwanese Hokkien automatic speech recognition (ASR), prior studies often treat tone sandhi as a major challenge under the assumption that models fail to process implicit phonological variations. However, our experiments on Taiwanese Hokkien reveal that speech foundation models actually handle tone sandhi variations effectively, and the real performance bottleneck stems from a localized confusion between these variations and retained citation tones. To address this, we propose T-SANDHI to explicitly decouple surface acoustics from underlying lexical intent on top of a frozen Whisper backbone. Using a lexicon-guided multi-task learning structure driven by text-derived pseudo labels, our lightweight hybrid injection module integrates independent citation and sandhi phonetic streams via dynamic gating. Extensive evaluation on the TAT-MOE corpus and two blind test sets demonstrates that this explicit disentanglement effectively resolves tonal mapping confusion, outperforming baselines with strict parameter efficiency.