Automatic speech recognition models vary widely for Iberian languages
Benchmarking Automatic Speech Recognition Tools for Iberian Languages
Computation and Language
Summary
Getting computers to understand spoken language, especially less common ones like Basque, is still tricky and varies a lot depending on the system used. The authors tested different speech recognition tools on five Iberian languages and found no one tool works best in every way. Some models are more accurate but slower, and others handle languages like Basque much worse because they haven't been trained enough. They also found the systems often performed differently for male and female voices, which means fairness in speech technology is still a problem.
What this means in practice
- •For speech technology developers: Select appropriate ASR models by balancing accuracy and efficiency across multiple Iberian languages for practical applications.
- •For multilingual service providers: Improve voice-enabled services by understanding ASR performance disparities, especially for low-resource languages and speaker sex biases.
Authors
Fernando López, Pablo Gómez, David Solans, Paulo Villegas, Jordi Luque
Abstract
Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficiency trade-offs are underexplored. We benchmark eleven systems, ten open-weight models and one commercial API, across five Iberian languages (Basque, Catalan, Galician, Portuguese, Spanish), with German and Turkish as controls. Evaluation uses an 85-hour dataset covering read speech, broadcast media, and audiobooks, assessing accuracy and efficiency via word error rate (WER) and real-time factors (RTF/RTFx). Results show no single model dominates: accuracy, efficiency, and language coverage present clear trade-offs. Low-resource languages, especially Basque, degrade significantly, highlighting the role of training coverage. We observe consistent sex disparities across most systems, highlighting fairness challenges in multilingual ASR. Overall, the benchmark provides practical guidance for real-world model selection.