Pruned speech models lose fairness across demographic groups
Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs
SoundComputation and Language
Summary
Speech recognition models are often compressed to run faster, but this study found that making these models smaller affects how well they understand different groups of people. The authors discovered that some demographic groups see their recognition accuracy drop more than others when the model is pruned. Even when adapting the model with additional training, groups that already had better results improved more, widening the fairness gap. These effects vary by language and dataset, so it's important to check errors for each group separately rather than relying on overall accuracy.
What this means in practice
- •For speech recognition engineers: Integrate per-group error metrics into model selection and deployment to ensure fairer performance across all demographics when pruning speech models.
- •For natural language processing teams: Assess compression impacts on accent and language variations by measuring group-wise word error rates for pruned speech models to avoid unfair performance drops.
Authors
Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi Shekhar
Abstract
Speech-LLMs are expensive to run, making compression important for real-world deployment. However, compressed models are usually selected using aggregate word error rate (WER), which can hide how pruning affects different demographic groups. In this work, we systematically study the effect of audio encoder pruning on SLAM-ASR for different demographic groups. Using the Fair-Speech and Common Voice datasets, we found that the pruning does not affect all demographic groups equally; the gap between best- and worst-performing groups increases in fold. These disparities appear across all three encoder scales, but only the largest model initially hides them behind aggregate WER. LoRA adaptation improves WER for every group, but benefits groups already performing well more strongly and widens for certain groups. On Common Voice English, Danish, and Dutch, accent gaps persist but do not clearly widen, showing that the fairness effects of pruning vary across datasets and must be measured directly. Our findings suggest that for pruned models, deployment decisions should include per-group WER, with the worst-performing group's error rate as an explicit criterion.