Word error rate often misaligns with human speech recognition quality judgments
Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
Computation and Language
Summary
Speech recognition systems are commonly judged by how many words they get wrong, but this method treats all mistakes as equally bad even if some don’t change the meaning. The authors created a new dataset to compare different ways of measuring errors against what people actually think about transcript quality. They found that word error rate is the least aligned with human opinions, while some newer metrics do better, though no one method is best in all cases. Simple character error rate also works surprisingly well and is cheap to compute. They suggest using character error rate as the main metric and newer semantic measures as a helpful complement.
What this means in practice
- •For speech recognition developers: Improve evaluation methods by prioritizing character error rate and semantic distance metrics to better match human judgments of transcription quality.
- •For translation software teams: Incorporate more human-aligned semantic evaluation metrics to enhance quality assessment of automatic speech-to-text outputs for multilingual users.
Authors
Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto, Petr Motlicek, Hyunggu Jung, Esaú Villatoro-Tello, Somang Nam
Abstract
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.