Chinese language models show bias with social media style expressions
Social Bias in Chinese Large Language Models under Social Media Expressions
Computation and Language
Summary
It can be tricky to find social biases in Chinese language models because they respond differently to normal and social media ways of saying things, like slang or emojis. The authors created special test questions that compare normal sentences, social media style sentences, and sentences with reversed identities. They found all tested models showed bias related to identity and struggled with sarcasm and stylistic changes. This means judging bias by just how believable a model finds a statement isn’t enough, especially when language is playful like on social media.
What this means in practice
- •For language model developers: Assess and improve Chinese large language model fairness by testing effects of social media style and sarcasm on identity bias.
- •For chatbot designers: Design chatbots that recognize and mitigate bias in Chinese social media style interactions, including slang and sarcasm.
Authors
Mo Zhou
Abstract
Existing Chinese social-bias benchmarks for large language models largely rely on normalized or template-based expressions, leaving the influence of internet slang, emoji, and sarcasm insufficiently examined. Safety alignment may further compress conventional plausibility ratings toward the lower end of the scale. This study constructs 73 paired probes across gender, region, and occupation. Each probe contains a neutral expression, a social-media-style expression, and a reversed-identity control. The task is reframed from direct plausibility judgment to social consensus estimation. Social-media expressions are divided into stylistic packaging, which preserves semantic polarity, and sarcastic packaging, which reverses surface polarity. Six Chinese-capable large language models are evaluated through paired differences measuring identity bias, stylistic packaging, and sarcasm shift. All models show significant identity-bias effects; five show significantly negative stylistic-packaging effects; and sarcasm shifts display a clear cross-model gradient. These findings demonstrate that expression form is not a transparent carrier in bias evaluation and that lower scores under sarcasm should not be interpreted directly as bias mitigation. The proposed paired design offers a measurement approach for Chinese social-media contexts while also revealing the construct boundary and prompt sensitivity of social consensus estimation.