Gender bias in large language models varies widely across vendors
Gender bias across LLMs is common and highly heterogenous
Computation and LanguageArtificial IntelligenceComputers and SocietyHuman-Computer Interaction
Summary
Gender biases in large language models (LLMs) are common, but the way these biases show up can be very different depending on the model. The authors studied ten models from nine different companies, testing how they attribute gender to typical ideas and how they judge moral dilemmas involving harm to men or women. Some models showed biases favoring men, others favoring women, and some showed no clear bias. This means checking for gender bias in LLMs should be continuous and cover many models, not just done once.
What this means in practice
- •For ai system builders: Continuously evaluate gender bias in multiple large language models to select fairer models for integration into tools with real-world impact.
- •For content moderation teams: Use insights about gender bias heterogeneity to design varied moderation strategies tailored to specific models’ tendencies toward biased outputs.
Authors
Edoardo Bolzoni, Valerio Capraro
Abstract
Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support tools with real consequences. Prior research has focused only on a small set of models, leaving open the extent to which gender biases are common and heterogeneous across LLMs. We address this gap across ten models released between April 2025 and June 2026, spanning nine vendors, using two paradigms: gender attribution to stereotyped phrases (Study 1) and moral judgment of abuse or torture against a woman or a man to prevent a catastrophic outcome (Study 2). In Study 1, two of ten models attributed masculine-stereotyped phrases to female writers more often than the reverse, while three models showed the opposite pattern. In Study 2, several models converged on a male-disadvantaging asymmetry that was directionally consistent with a documented human tendency to protect female targets from harm, though the specific conditions under which this asymmetry emerged varied by model; three other models, by contrast, showed no variation across conditions. These results indicate that gender-related biases are common in LLMs. Their direction and magnitude, however, are highly heterogeneous, to the point that some models behave in diametrically opposite ways to others. Bias auditing should therefore be treated as an ongoing, multi-vendor process, rather than a one-time assessment.