Translated benchmarks work best when content matches local knowledge
Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?
Computation and Language
Summary
Many language tests are originally created in English and translated into other languages to check how well AI models understand different languages. The authors studied if these translated tests measure the same skills as tests made specifically for those languages. They found that translated tests work better when the questions overlap with local school materials, but they don’t capture local facts or knowledge as well. This means that using translated tests alone might not fully show how well AI understands a particular culture or language.
What this means in practice
- •For language model evaluators: Select or design evaluation tests that share content overlap with local education materials for more accurate model assessment in non-English languages.
- •For multilingual ai developers: Use findings to decide when translated benchmarks can reliably measure model knowledge and when local data is needed for training and evaluation.
Authors
Klaudia-Doris Thellmann, Bernhard Stadler, Dan Saattrup Nielsen, Michael Färber, Jens Lehmann
Abstract
Translated versions of English-origin benchmarks such as MMLU are widely used to evaluate large language models in non-English languages, yet it remains unclear whether they measure the same constructs as natively created benchmarks. We frame this as a proxy validity question and compare DeepL-translated MMLU proxies against six native benchmarks in seven pairings across five Nordic and Baltic languages, evaluating 29 models on each pairing. Beyond ranking agreement, we test incremental proxy validity: does the translated score add predictive value for native performance after controlling for model size and language proficiency? Proxy validity varies substantially across benchmark pairings. Applying the same robustness criteria to all seven pairings, we find four robust effects, two marginal effects, and one null result. The strongest evidence occurs for curriculum-overlap benchmarks such as Skolprov ($ΔR^2$=.128, 95% CI [.038,.253]), whereas translated MMLU provides no significant incremental value for Sweden-specific factual knowledge in SwedishFacts (p=.242). Because SwedishFacts is also the only open-ended benchmark, we cannot fully separate the effects of local content and response format. Overall, the results indicate a clear pattern: translated benchmarks provide a better proxy when their source content overlaps with the native evaluation target, while locally anchored knowledge is less reliably captured.