Large language models show varied stereotype biases across languages and contexts
From Language to Country to Valence: A Multilingual Audit of Stereotype Bias Across LLMs
Computation and Language
Summary
Large language models are used in many important areas like education and hiring, but they can have social biases. This paper studies how these biases change depending on the language and cultural context by using Reddit posts in four languages. The researchers found that biases differ by stereotype category and language, and that some commercial models perform worse than open models. Their results show that average bias scores hide important details, suggesting closer checks are needed before using these models widely.
What this means in practice
- •For ai deployment teams: Evaluate language models for specific language and cultural bias patterns before using them in sensitive settings.
- •For human resources software developers: Improve bias detection in recruitment tools by incorporating multilingual and context-sensitive evaluations offered by this study.
Authors
Zihao Sun
Abstract
Large language models (LLMs) are increasingly considered for education, recruitment, workplace decision support, and other consequential applications, making pre-deployment social-bias detection essential. Yet major benchmarks are English-centred or translated, focus mainly on negative stereotypes, and may miss linguistic and cultural variation. We investigate gender, racial, and religious bias using 12,000 locally written Reddit posts in English, Spanish, French, and German. Only search terms are translated, preserving local context. The dataset is balanced by language and category; English gender is additionally balanced by stereotype valence. Three open-weight and three proprietary API-based models are evaluated with a two-blank design that reduces grammatical-gender cues. Religion has the highest absolute choice imbalance in every model-language condition. Spanish is highest for gender, whereas German is highest for race and religion. The evaluated API models score below the open-weight models throughout. Negative English-gender items exceed positive items; Australian and Mexican contexts exceed their Canadian and Spanish counterparts. These descriptive differences measure choice concentration, not stereotype endorsement or causal country effects. The crossed patterns show how aggregate scores can conceal category- and context-specific behaviour, motivating closer inspection before deployment.