Acmite reduces gender bias in language models using concept guidance
Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Information
Computation and Language
Summary
Large language models sometimes repeat unfair gender stereotypes from the data they learned from. The authors propose Acmite, a method that identifies the ideas behind stereotypes as concepts and reduces their influence without changing unrelated parts of the model. Acmite works by training a small addition to the model that only activates when the input is close to stereotype concepts, helping the model avoid bias while keeping it accurate on other tasks. Tests show Acmite effectively lowers gender bias and maintains overall performance.
What this means in practice
- •For ai application developers: Integrate Acmite to reduce gender bias in conversational AI responses without harming general language capabilities.
- •For chatbot platform engineers: Use Acmite’s lightweight adapter to selectively activate debiasing only when inputs resemble stereotype concepts, preserving efficiency and accuracy.
Authors
Tian Lan, Xiaoqing Cheng, Han Zhang, Jiang Li
Abstract
Large language models (LLMs) can reproduce social stereotypes from their training data, motivating extensive research on model debiasing. However, existing methods often rely on explicit biased examples or predefined group-term substitutions, making them sensitive to wording and less effective at capturing stereotype concepts shared across diverse contexts. More importantly, they typically suppress biased outputs without explicitly modeling the statistical dependence between model outputs and the underlying stereotype concepts. We propose Acmite, a lightweight concept-guided framework for targeted and selective debiasing. Acmite represents stereotypes as structured semantic concepts and uses maximal marginal relevance (MMR) to select diverse concepts for debiasing. Inspired by mutual information minimization, it approximates this dependence with token-level KL divergence while preserving task semantics. A lightweight LoRA adapter is trained with the base model frozen and activated at inference time only when the input is sufficiently similar to stereotype-related concepts; otherwise, the original model is used directly. We evaluate Acmite on BBQ, CrowS-Pairs, and StereoSet, and assess general capability preservation on ARC-Challenge, GSM8K, and PIQA. Experiments across three LLMs show that Acmite effectively mitigates gender bias across complementary evaluation formats while maintaining competitive performance on bias-unrelated tasks. Anonymous code and data are available at https://anonymous.4open.science/r/Acmite-18E2/.