Training seed effects outweigh compression in speech recognition fairness
Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation
Computation and Language
Summary
Fairness differences in speech recognition systems are often measured using a single training run, but this can be misleading. The authors found that changing the random starting conditions (training seeds) causes bigger changes in how fair the system is across different ethnicities than changing how much the audio is compressed. Even when using more data, these training seed effects remain important. This means that assessing fairness should consider variability across multiple training runs to get a clearer picture.
What this means in practice
- •For speech recognition engineers: Account for training seed variability when tuning speech models to better assess and improve demographic fairness across runs.
- •For machine learning operations teams: Use multiple training seeds to evaluate fairness metrics reliably when deploying speech recognition systems at scale.
Authors
Srishti Ginjala, Eric Fosler-Lussier, Srinivasan Parthasarathy
Abstract
Demographic fairness gaps in automatic speech recognition are almost always reported from a single training run. We fine-tune the Q-former projector and LoRA adapters of a speech LLM at five audio compression factors and six random seeds, holding the encoder, base decoder, data and decoding fixed, and evaluate every run on Common Voice and Fair-Speech. At 460 h of clean LibriSpeech, the seed moves fairness metrics more than compression does on most demographic axes. A balanced 3x3 decomposition attributes 85.3% of the variation in Fair-Speech ethnicity normalized gap to the seed against 8.3% to compression (p = 0.009), though compression explains more on age and gender. Held-out LibriSpeech word error rate spreads by 0.04 points across those seeds while Common Voice spreads by 8.57, so these are not failed runs, and the effect survives controlling for accuracy and dropout. Scaling and diversifying the adaptation set to 960 h damps the effect but does not remove it. On Fair-Speech ethnicity, two single-run systems must differ by more than 0.30 in normalized gap to exceed seed variability.