Korean benchmark suite evaluates AI model skills beyond translations
KBS: A Korean Benchmark Suite for Foundation Model Evaluation
Computation and Language
Summary
Measuring how well AI understands Korean is tricky because simple translations don’t fully capture language nuances. The authors created a detailed Korean Benchmark Suite (KBS) that tests different abilities with carefully designed Korean tasks rather than just translated ones. They tested various AI models and found that scores depend heavily on how tests are made and scored, as well as how models are prompted. This means that when testing AI in Korean, you need to share both the scores and the exact test setup to get a clear picture.
What this means in practice
- •For natural language processing teams: Use the suite to evaluate Korean language understanding of AI models with tasks tailored beyond simple translations.
- •For multilingual model developers: Test model performance in Korean more accurately to guide improvements specific to Korean capabilities and data.
Authors
Yeonkyoung So, Jongmin Kim, Sungmok Jung, Gyuseong Lee, Sangho Kim, Jongyeon Park, Joonhak Lee, Seho Pyo, Gyeongje Cho, Seorin Kim, Jisoo Kim, Suyoung Park, Hyunji M. Park, Yelim Ahn, Yeongho Seo, Jaejin Lee
Abstract
We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities across 9,396 items. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation-only approach to benchmark adaptation. We evaluate Korean-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post-training. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them.