Representation steering does not reliably reduce speech recognition bias
A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models
Computation and Language
Summary
Speech recognition systems often make more mistakes with some groups of speakers than others. The authors tested whether changing internal model features linked to speaker traits like gender, age, or accent can fix these errors. They found that even when these traits are easy to detect inside the system, nudging the model to ignore them hardly improved error rates. Sometimes making the model pay more attention to these traits actually made errors worse. This means just spotting these traits inside a model is not enough to fix fairness problems.
What this means in practice
- •For speech system engineers: Identify limits of adjusting internal speech model features to reduce recognition errors across speaker groups.
- •For machine learning fairness teams: Evaluate bias intervention methods jointly at representation and task outcome levels for more reliable fairness improvements.
Authors
Nicolas Bourrel, Abderrahmane Issam, Gerasimos Spanakis
Abstract
Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representations. We ask whether speaker-linked attributes that are linearly readable from pretrained ASR encoders yield useful directions for reducing group word-error-rate (WER) gaps. Across Whisper-medium, HuBERT-large, and Wav2Vec2-large on Common Voice and the Speech Accent Archive, we probe every encoder layer for metadata-derived sex/gender, age, and native/accent labels; construct centroid and probe-derived directions; inject them at selected layers; and compare downstream probe trajectories with matched WER changes. Sex labels are highly decodable (best macro-F1 0.924--0.941), native/accent labels are also above chance (0.544--0.696), and age is weaker (0.354--0.397). Of 22 post-selected reruns, nine have 95% paired-bootstrap intervals entirely below zero, yet every absolute source-group WER reduction is below 0.7 percentage points. Conversely, a local target-class probe rate can rise from 8.09% to 99.87% while WER worsens. Linear readability is therefore neither evidence of causal use nor a reliable mitigation method. Our results motivate evaluating speech-bias interventions jointly at representation, propagation, and task levels.