CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms
Computation and Language
Summary
The authors studied new Chinese words (neologisms) that use special language tricks like replacing sounds with numbers (e.g., 886 for 'bye-bye') or breaking down characters visually. They created CNeo-Bench, a collection of almost 5,000 such words, categorized by how they’re made, and tested 18 large language models (LLMs) on understanding and manipulating these words. They found most models struggle, especially with accurately restoring the original form rather than just giving synonyms. Even giving models some examples helped but didn’t fix all errors, showing this is still a hard problem.
Authors
Kaiyan Zhao, Zhongtao Miao, Zheyong Xie, Shaosheng Cao, Yoshimasa Tsuruoka
Abstract
Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character decomposition that are rare in other languages. We introduce CNeo-Bench, a benchmark of 4,759 such neologisms with reference definitions, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression. CNeo-Bench is paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whether it can operate on its underlying mechanism. Evaluating 18 LLMs, we find that Chinese neologisms remain an open challenge; most models fall below 40\% on definition generation, and on several subcategories a systematic recognition-manipulation gap emerges: models describe neologisms correctly but, in source-form restoration tasks, substitute a semantic equivalent (paraphrase) for the source form rather than producing the source form itself. A few-shot analysis on 1,058 hard items shows that in-context examples can solve many difficult cases, but leave a noticeable portion of errors remaining, indicating challenges beyond prompting alone can address.