Token-level tolerance improves speech recognition accuracy across languages
A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition
Computation and Language
Summary
Speech recognition systems usually learn by assuming one exact transcript for what is said, but real spoken words often vary slightly in spelling or pronunciation. The authors developed a way to let the training ignore small unsure parts at the level of individual tokens instead of entire words, which helps the system learn better. Testing on many languages and datasets showed this approach reduces errors significantly. They also changed how the system adapts during training, making it less dependent on how long it trains. Their method better targets parts of speech where transcripts disagree, improving recognition accuracy.
What this means in practice
- •For speech recognition developers: Reduce errors in speech-to-text systems by training with token-level tolerance to ambiguous transcript parts across many languages and datasets.
- •For multilingual voice assistant teams: Improve transcription accuracy in voice assistants by using hybrid token- and word-level relaxation during training to better handle transcript uncertainties.
Authors
Saurabh Kumar, Diptiman Mohanta, Prasanta Kumar Ghosh
Abstract
Automatic speech recognition is typically trained assuming that the reference transcript is the only valid labeling of an utterance, yet even nominally verbatim transcripts contain localized differences in pronunciation, spelling, or lexical realization that the acoustics do not uniquely determine. Omni-temporal Classification (OTC) tolerates such noise by adding wildcard paths to the connectionist temporal classification (CTC) alignment graph, but its word-level arcs are too coarse, since bypassing one unsupported token discards supervision for the whole word. We move wildcard arcs to token granularity so unsupported tokens can be bypassed while the rest of the word stays supervised, and we combine token- and word-level arcs as complementary escape paths. Across 19 languages and three corpora, token-level OTC improves over CTC on all 25 tasks. We also replace epoch-indexed relaxation of the wildcard weights with a predictive-entropy-indexed schedule, which performs comparably while reducing dependence on training length. Combining this schedule with the hybrid graph gives the lowest mean word error rate (WER) on every corpus and a 9.45% average relative WER reduction over CTC. Independent validator transcriptions show that token-level models place significantly more wildcard-bypass probability than CTC on disputed characters, indicating that token-level tolerance targets localized transcript ambiguity.