Number formatting can distort speech recognition error scores
The Hidden Cost of Digits: Number Normalization and WER in ASR Systems
Computation and Language
Summary
Automatic speech recognition systems sometimes write numbers as digits instead of words, which makes it harder to compare how well different systems work. The authors studied Polish, a language where words change form depending on context, to see how this affects error measures. They found that ignoring number formatting can make error rates look much worse, by over 2 percentage points. This difference is often bigger than gaps seen between competing speech recognition systems.
What this means in practice
- •For speech recognition developers: Improve evaluation fairness by including number normalization when comparing ASR transcripts, especially for languages with complex word forms.
- •For language technology evaluators: Refine benchmarking procedures to better handle numerical expressions in transcripts, yielding more accurate system comparisons.
Authors
Stanisław Kacprzak, Mieszko Fraś
Abstract
Modern automatic speech recognition (ASR) systems trained on extremely large datasets can produce transcripts with numbers written in Arabic numerals. This creates a need for fair comparison with models that output verbatim texts and proper processing of reference transcripts. Popular approaches often reduce text normalization to lowercase and remove punctuation, with no additional normalization applied to languages other than English. In this work, we analyze the impact of normalization of numerical expressions in the evaluation of ASR systems in various languages, using Polish as an example of a highly inflective language. We perform experiments on VoxPopuli and The Polish Parliamentary speech datasets and estimate word error rate (WER) differences for different text normalization approaches. We show that the difference due to the lack of number normalization in WER may be substantial - more than 2 percentage points, and often higher than the differences between systems in popular multilingual benchmarks.