Large language models face challenges matching human translation quality judgments
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Computation and Language
Summary
Checking how good machine translations are is usually done by humans, but it can be slow and expensive. This paper tests if big AI language models can do this job instead and how well they agree with human judges. The authors found that sometimes these AI models agree more with humans than humans do with each other, but mostly, both humans and AI have trouble being consistent. For example, humans struggle with detailed error categories or rare languages, while AI has difficulty spotting small mistakes or identifying error locations. The study suggests working together with humans and AI might improve translation quality checks.
What this means in practice
- •For machine translation engineers: Use combined human and LLM feedback to improve translation quality assessment pipelines across many language pairs and domains.
- •For content localization teams: Identify when automated translation quality checks might be unreliable, especially for minor errors or less common languages, to plan human reviews accordingly.
Authors
Hala Almaghout, Christian Federmann, Qin Gao
Abstract
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.