Large language models improve machine translation quality checks efficiently
Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Computation and Language
Summary
Checking how good machine translations are usually requires expert humans, which can be slow and expensive. The authors studied how different ways of using large language models (LLMs) affect their ability to evaluate translations. They found that how instructions and examples are given to the models changes their judgments, and combining different LLMs works better than using just one. Their best setups perform as well as experts while being much cheaper and faster.
What this means in practice
- •For machine translation engineers: Automatically evaluate translation quality with expert-level accuracy at substantially lower costs using combined LLM evaluators.
- •For software localization teams: Use reliable automated assessments of translation fluency and accuracy to speed up content localization processes.
Authors
Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev, Stepan Vysotsky, Nikolay Karpachev
Abstract
LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood. We present a systematic study of how instruction design, few-shot demonstrations, backbone model choice, and multi-step workflows affect span-level and ranking-based MT evaluation. We collect new expert RATE annotations on WMT24 and WMT25 English-Russian translations, with transfer validation on MQM- and ESA-annotated benchmarks across five additional language pairs. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles (recall-, precision-, or ranking-oriented), and fusing outputs across diverse backbones outperforms single-evaluator multi-call setups. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs. We synthesize these findings into ORBIT, a family of LLM-based MT evaluators. Our strongest configurations achieve expert-level performance in error span detection and in Accuracy and Fluency ranking at roughly 20x lower cost than expert human annotation.