Audio language models reduce gender bias with adversarial training
Bias Mitigation in Audio-Enabled LLMs through Adversarial Training and Distillation Methods
Computation and LanguageSound
Summary
Audio-enabled language models let people talk directly to them, capturing spoken words and non-verbal sounds. However, these models can guess people's gender from speech, which may lead to biased responses. The authors found ways to train these models so they forget how to detect gender, avoiding bias, without losing their overall ability. They also created versions that keep gender recognition but avoid bias unless it’s specifically asked about. They tested these methods on three different models with good results at removing gender bias.
What this means in practice
- •For voice assistant developers: Build voice assistants that avoid unintended gender bias by unlearning gender perception from audio inputs.
- •For customer support platform teams: Create chat systems that understand speech without producing gender-biased responses, improving fairness and inclusivity.
- •For call center software vendors: Offer AI-powered voice interfaces that subtly detect gender when needed but avoid gender bias in general interactions.$Commercial implications: This enables gender-aware voice systems that deliver unbiased responses, appealing to businesses focused on ethical AI and diverse customer bases.
Authors
Adnan Al Ali, Jindřich Libovický, Anne Lauscher
Abstract
Audio-enabled large language models (AuLLMs) allow users to interact with them simply by talking to them, without the need for an intermediate transcription. This expands the model's accessibility and capabilities by encoding non-verbal cues. However, using non-verbal cues may lead to biased behaviour based on the attributes inferred (or assumed) from them. In this work, we focus on the gender attribute and show that models' perception of it can be unlearned using adversarial methods without a loss in performance on other benchmarked tasks. Such models are useful in situations where gender inference is unwanted. However, in other practical settings, gender perception (without bias) may be desired.As a less radical measure, we distil the gender-agnostic model, creating a model that perceives gender but responds similarly to the gender-agnostic teacher, unless explicitly asked.We apply our newly proposed methods to three AuLLMs and obtain satisfactory results in the gender-unlearning part and partially satisfactory results in the distilling part.