OpenTWBench measures large language models on Taiwan knowledge
OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score
Computation and Language
Summary
Measuring how well language AI models know Taiwan-related facts is tricky. The authors created OpenTWBench, a test suite with questions from Taiwan's national exams and everyday Taiwan topics. They show how the way answers are extracted or how the test is set up can dramatically change model scores. This new benchmark and its open data let others fairly compare AI understanding of Taiwan knowledge.
What this means in practice
- •For nlp engineers: Evaluate and improve large language models’ knowledge specifically related to Taiwan using standardized, openly available benchmarks.
- •For ai platform operators: Integrate a Taiwan-domain benchmark into model quality control pipelines to track model progress on regional knowledge coverage.
- •For language learning app developers: Use Taiwan-focused factual question data to tailor AI-driven educational content for users interested in Taiwan culture and knowledge.$Commercial implications: Enables targeted content creation for Taiwan-related learning apps to enhance commercial user engagement.
Authors
Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang, Yu-Wei Liu, Che-Yu Wu, Po-Feng Lee
Abstract
We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai