Chinese internet buzzwords challenge large language models across languages
CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords
Computation and LanguageArtificial Intelligence
Summary
Chinese social media uses many special words that are tricky because their meanings are deeply tied to Chinese culture and language. The paper studies if advanced language AI programs can understand these special Chinese words when translating or explaining them in English. To test this, the authors created a big set of Chinese buzzwords with detailed English explanations, equivalents, and safety labels. They found that current AI models still have a hard time with these words, especially when the meanings are not direct and depend on culture or humor. This shows that AI still struggles with language that is strongly shaped by specific cultures.
What this means in practice
- •For content moderation teams: Use the benchmark to improve AI systems that detect harmful or offensive content hidden in culture-specific Chinese buzzwords when monitoring social media.
- •For multilingual chatbot developers: Develop chatbots that better understand and respond to Chinese internet slang when interacting with bilingual or English-speaking users.
Authors
Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu, Fanliang Bu
Abstract
Chinese social media has generated a vast and continually evolving lexicon of internet buzzwords whose meanings are often non-literal and deeply rooted in local cultural and pragmatic contexts. Existing research has primarily focused on interpreting these buzzwords within Chinese, leaving largely unexplored whether LLMs can transfer such culturally grounded knowledge across languages and accurately convey the intended meanings in English. This cross-lingual capability is also critical for safety, as harmful expressions may obscure their offensive content through culture-specific homophony, euphemism, irony, or coded language. In this paper, we investigate the ability of advanced LLMs to understand Chinese internet buzzwords across languages. To this end, we introduce CIBuzzBench, the first benchmark for cross-lingual Chinese-to-English understanding of Chinese internet buzzwords. CIBuzzBench comprises 3,001 Chinese internet buzzwords annotated with English meaning explanations, English equivalents, category labels, and harmfulness labels. Based on these annotations, we design three evaluation tasks: Meaning Explanation, Cross-lingual Equivalent Matching, and Culturally Grounded Harmfulness Detection. We evaluate representative state-of-the-art proprietary and Chinese LLMs under both English- and Chinese-prompting settings. Our results show that LLMs continue to struggle with the cross-lingual understanding of Chinese internet buzzwords, particularly in fine-grained non-literal interpretation, robust equivalent matching under option perturbations, and calibrated harmfulness detection. These findings highlight the persistent challenges posed by culturally grounded language phenomena for multilingual LLMs and safety-oriented evaluation. The dataset and code are available at https://github.com/SuperYFan/CIBuzzBench.