Audio models bias found and reduced by new voice testing technique
When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models
SoundComputation and Language
Summary
Speech recognition systems make more mistakes for people with certain voices, like Black speakers or those with accents. The authors created TRIAD, a way to test these systems by playing voices that are controlled to sound like different ages, genders, and accents, making it easier to find where biases happen. They developed mathematical tools to measure how much the systems mix voice traits with the meaning of words, which causes unfair errors. Then, they built ORCA, a method to reduce this mix-up, which greatly lowers these biased errors.
What this means in practice
- •For speech technology developers: Identify and reduce demographic bias in speech recognition models using TRIAD and ORCA methods to improve fairness across voices.
- •For voice assistant engineers: Improve accuracy of voice assistants for diverse speakers by applying contour controls and fairness audits from TRIAD and ORCA techniques.
Authors
Kian Shamsaie, Iman Modarressi
Abstract
Speech technology penalizes some voices: recognition errs nearly twice as often for Black speakers, and accuracy declines for second-language accents and older speakers. We introduce TRIAD, an audit grid crossing 120 texts, 24 rendered demographic voice profiles (gender, age band, accent), and ten expressive styles via controllable text-to-speech, isolating perceived demographic attributes from content and affect. For ten open-weights encoders we define axis-fidelity functionals, principal-angle leakage between axis subspaces, and group-conditional gaps; a proposition proves that average probe disparity grows with the same aggregate voice-semantic leakage $Λ$ we measure, and a corollary shows that peak leakage forces worst-case disparity inside an active region. The measured mean-square probe disparity tracks $Λ$ (Pearson r = 0.93), and a black-box protocol exposes the same signature in two closed-source models. ORCA, an adapter combining axis-specific contrastive heads, an orthogonality penalty, and group-balanced sampling, cuts leakage 72% and roughly halves the gaps.