Beyond Surface Cues: Disentangling Sociocultural Signals in Multilingual LLMs
2026-08-24 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how big language models (LLMs) in English, French, and Chinese handle cultural and social bias in their responses. They showed that some signals, like names or wording related to a language, can mislead researchers into thinking the model truly understands culture when it might just be picking up on obvious cues. By carefully removing these cues and using human checks, the authors found that biases and cultural patterns appear differently depending on the language and task. Their work helps separate real cultural understanding from simple tricks in multilingual AI outputs.
Multilingual LLMsCultural groundingSocial biasIdentity cuesCross-cultural patternsHuman validationTranslation effectsBias auditLanguage modelsAutomated vs human ratings
Authors
Yuanjun Feng, Tanzhou Liu, Stefan Feuerriegel, Yash Raj Shrestha
Abstract
Multilingual LLM outputs can vary across sociocultural contexts. However, evidence of cultural grounding can be misleading: identity labels may be inferred from explicit or indirect textual cues, while names and wording can reveal the source language. Treating all these signals as evidence of cultural grounding may obscure potential biases. We present a human-validated, multi-agent audit that separates three questions: whether outputs reproduce social biases, whether identity groups are represented differently, and whether outputs reflect cross-cultural patterns. The study analyzes 89,253 outputs from 12 LLMs in English, French, and Chinese, spanning 18 occupations and three task conditions. We find that bias representation varies systematically across languages and tasks. Removing direct identity cues sharply reduces identity-label prediction in English and Chinese, but has a much smaller effect in French. Across all language-genre settings, the cultural context associated with the source language receives the highest average relevance score, with moderate agreement between automated and human ratings. However, the ability to identify the source language drops substantially after translation and again after masking names. Without these controls, multilingual audits may mistake surface cues for cultural understanding, leading to misleading conclusions about cross-cultural variation and bias. Our audit offers a practical framework for separating such shortcuts from more meaningful cross-cultural patterns.