Text to speech errors often come from automatic speech recognition mistakes
Measuring the Verifier's Floor: How Much Apparent Rare-Word Text-to-Speech Error Is ASR Error?
SoundComputation and Language
Summary
When computers read text aloud and their speech is checked by other computers that try to write down what was said, errors can happen in either step. The authors found that many errors thought to be from text-to-speech systems are actually from the speech recognition systems mishearing rare words. By comparing multiple speech recognizers and checking actual human speech recordings, they showed that recognizing rare words is challenging for these systems. This means evaluating speech systems by transcription can confuse real speech errors with recognition mistakes.
What this means in practice
- •For speech technology developers: Improve evaluation methods to distinguish true TTS errors from ASR recognition errors, especially on rare words, for more accurate TTS system testing.
- •For customer service automation teams: Refine speech evaluation processes to better identify when speech recognition errors affect automated voice systems, enhancing error diagnosis and system improvements.
Authors
Kirill Borodin, Vasilii Kudryavtsev, Nikita Vasiliev, Grach Mkrtchian
Abstract
Text-to-speech (TTS) is usually evaluated by transcribing its output with an automatic speech recognition (ASR) system, so the word error rate also counts recognizer errors. We measure these errors on human recordings of the test sentences. On LibriSpeech and FLEURS, the evaluation recognizer misses 33-67% of very rare words even in human speech, and all three recognizers miss 22-42%. Checking every word with two more recognizer families lowers the apparent TTS error on words that at least one recognizer hears two- to three-fold (very rare LibriSpeech words: from 10-13% to 3-5%), yet very rare words remain the hardest. Ignoring words the recognizer misses in human speech is biased: the other recognizers find TTS errors far more often on these words (4-6% vs. 0.2-0.3%) and overturn about half of the errors this mask keeps. As half of the remaining adjudicated errors may be spelling variants, a listening test is needed.