African speech dataset reveals recognition biases in low resource languages
AfriBED: a Speech Recognition Bias Evaluation Dataset for Low Resource African Languages
Computation and LanguageArtificial Intelligence
Summary
Speech recognition systems sometimes understand some voices better than others, especially based on gender or speaker type. The authors created a new dataset with recordings from two African languages, Bemba and Nyanja, to test if these systems are fair across different speakers. Their tests showed the dataset can help find biases in speech recognition for these languages. This helps improve speech technology so it works well for everyone, not just some groups.
What this means in practice
- •For speech technology developers: Evaluate and reduce gender and speaker-type biases in African language speech recognition systems using the AfriBED dataset.
- •For language technology project teams: Test speech recognition fairness in low resource African languages by leveraging the AfriBED dataset in development workflows.
Authors
Claytone Sikasote, Hussein Suleman, Jan Buys
Abstract
Despite advancements in automatic speech recognition systems (ASR), current systems exhibit speaker-attribute bias, i.e., their speech recognition performance is biased against certain speakers based on their demographic attributes such as gender, accents, and speaker type. However, accurately assessing speaker attribute bias in a given ASR system requires the availability of a bias evaluation dataset with comparable speech across subgroups of a speaker attribute. In this paper, we introduce African speech Bias Evaluation Dataset (AfriBED), a novel speech recognition bias evaluation dataset for two low resource African languages, Bemba and Nyanja, to detect and estimate gender and speaker-type bias for ASR systems. The dataset comprises 9 hours of speech with 3928 utterances recorded by 80 speakers for Bemba, while the Nyanja set includes 6260 utterances with approximately 18 hours of speech recorded by 127 speakers. The results of our baseline experiments demonstrate that the datasets are useful for examining gender and speaker-type attribute bias in the target languages.