KoNeoBench tests large language models on Korean new words
KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms
Computation and LanguageArtificial Intelligence
Summary
Language changes all the time with new words appearing, but many computer language tests only check old or common words. The authors created KoNeoBench, a new test to see how well language models understand recent Korean words from news since 2020. They collected expert-reviewed examples, word parts, and meanings for 1,785 new Korean words to build the test. Their results show that current language models struggle to break down these new words and explain their meanings accurately.
What this means in practice
- •For natural language processing engineers: Improve Korean language models by testing and refining them with recent words and meanings.
- •For machine translation developers: Enhance Korean translation quality by evaluating models on new Korean vocabulary not covered before.
Authors
Soha Lee, Soojin Lee, Heesung Yang, Hyunju Song, Hyunji Lee, Jinsan An, Jeongwan Shin, Jin Hyun Park, Jun Lee, Hyeyoung Park, Kilim Nam
Abstract
Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary and therefore provide limited coverage of such recent lexical change, and their English-oriented design makes it difficult to assess the typological properties of Korean, in which content words combine productively with functional morphemes. In this paper, we introduce KoNeoBench, a benchmark for evaluating LLMs' understanding of Korean neologisms. KoNeoBench is built on 1,785 Korean neologisms attested in online news since 2020 and curated through expert lexicographic review. Each entry provides usage examples, word-formation analyses, and dictionary-style definitions. Based on this resource, we define four tasks and report results on recent models, together with a human baseline. Our experiments show that current LLMs exhibit clear limitations in recovering source components, distinguishing semantic categories, and generating accurate definitions. These results reveal specific aspects of recent Korean lexical change that remain challenging for current LLMs. KoNeoBench is available at https://github.com/bcmilab/ko-neobench/ .