Large language models show bias in detecting hate speech about women
Same Sentence, Different Construct: Identity-Swapped Rationales Reveal Measurement Non-Invariance in LLM Hate Speech Classifiers
Computation and Language
Summary
Large language models are used to detect hate speech, but they don't always work the same way for different groups. The authors tested whether these models label hate speech equally when the target group changes, like replacing a group's name in the same sentence. They found that some models miss hateful statements about women more often than about other groups and explain their decisions differently. This suggests these models are not always fair or consistent, even if their overall accuracy seems high.
What this means in practice
- •For content moderation teams: Improve hate speech detection by identifying and correcting group-specific biases in language model classifiers.
- •For ai fairness engineers: Use identity-swapped templates and rationale analysis to evaluate and reduce bias in large language model measurement tools.
Authors
Wenchang Wang, Yue Song
Abstract
Large language models are increasingly used as measurement instruments for socially consequential constructs such as hate speech, and are typically validated by their aggregate agreement with human labels. A valid instrument must also apply the same construct to every group it measures. We test this requirement using 421 \hc{} templates that are instantiated identically for seven protected groups, so that only the identity term varies. Four LLM classifiers each exceed 0.95 balanced accuracy, yet within identical templates two of them miss hateful statements about women at markedly higher rates than the same statements about other groups (excess false-negative rates of 8.3 and 4.4 percentage points), a gap first reported for fine-tuned classifiers in 2021. We show how the construct shifts. Comparing rationales within each template, we find that models invoke protected-group status for other groups but recast the identical statement about women as personal dislike or as aimed at an individual, in 22 of 30 missed templates for one model. Adding the benchmark's own definition, which lists sex as a protected characteristic, does not close the gap, and the contrast persists. Apparent false-positive disparities, by contrast, largely vanish under identity control and trace to reclaimed slurs present only for some groups. Errors shared by all models concentrate where model and benchmark definitions diverge, and pairwise counterfactual accuracy adds little beyond endpoint accuracy. We argue that validating LLM measurement requires identity-controlled tests of measurement invariance that examine both decisions and the criteria models state for them.