Dense acoustic modeling improves pronunciation assessment and segmentation
Subphonetic Acoustic Modeling via Optimal Transport for Pronunciation Assessment
Sound
Summary
Pronunciation assessment needs sounds to be analyzed with precise timing and useful detail. The authors show that existing methods either miss fine timing or don’t capture subtle sound clues well. They introduced a new acoustic model that looks inside each sound to track smaller units continuously, improving the accuracy of timing and mispronunciation detection. This new model works better on different types of speech and helps in automatically judging how well someone speaks a language.
What this means in practice
- •For language learning app developers: Integrate a frame-level acoustic model to provide more accurate timing and detail for automatic pronunciation feedback in apps.$Commercial implications: Enables selling enhanced pronunciation evaluation features that better detect timing and mispronunciations for language learners.
- •For speech therapy technology teams: Use improved segmentation and detection of pronunciation errors to develop better tools for diagnosing and tracking speech disorders.
Authors
Haopeng Geng, Jiun-Ting Li, Daisuke Saito, Nobuaki Minematsu
Abstract
Pronunciation assessment requires acoustic evidence that is temporally precise, diagnostically meaningful, and faithful to the learner's actual production. However, existing acoustic models often struggle to provide recognition and segmentation evidence simultaneously. CTC-based phone recognizers can predict phone sequences flexibly, but their sparse and peaky posteriors often miss phone boundaries and fine-grained pronunciation cues. In contrast, text-dependent forced aligners provide reliable temporal information when transcripts are available, but are not directly applicable to reference-free pronunciation analysis. In this work, we propose a topology-aware frame-wise acoustic model that learns dense ordered state posteriors within each phone. The key idea is to recover phone-internal state structure in a neural acoustic model by combining ordered subphonetic states with optimal temporal transport classification (OTTC). This combination encourages dense monotonic frame-level state discrimination while preserving phone recognition ability. Experiments on read, spontaneous, and L2 speech show improved segmentation over neural baselines with competitive recognition performance. Downstream evaluations further show gains in mispronunciation detection and automatic pronunciation assessment. Probing analysis suggests that the learned states capture phoneme-dependent acoustic structure rather than arbitrary frame-level distributions.