Machine translation evaluation improved with expert scoring method
Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems
Computation and Language
Summary
Machine translation systems are getting so good that it’s hard to tell their mistakes using old evaluation methods. The authors found that usual tests miss small errors because of inconsistent judgments and limited focus on accuracy over the flow of language. They created a new dataset scored by professional linguists who took their time, along with a simpler and clearer scoring system called RATE. Their results show that some machines now translate more accurately than humans but still don’t sound as natural, a difference older tests couldn’t reveal.
What this means in practice
- •For machine translation developers: Use the RATE protocol and professional annotation dataset to improve evaluation of translation accuracy and fluency separately for better system tuning.
- •For localization teams: Access refined human evaluation tools to identify subtle translation issues impacting naturalness, improving the quality of localized content.
Authors
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
Abstract
As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We critically examine limitations in current gold-standard approaches (MQM and ESA), including inconsistencies from variable annotator expertise, excessive categorization complexity, coarse severity granularity, accuracy bias over fluency, and time constraints. To address this issue, we introduce a high-quality dataset consisting of human evaluations for English-Russian translations from WMT24, created by professional linguists. We show that expert assessments without time pressure yield substantially different results from standard evaluations. To enable consistent and rich annotation by these experts, we developed the RATE (Refined Assessment for Translation Evaluation) protocol. RATE provides a streamlined error taxonomy, expanded severity ratings, and multidimensional scoring balancing accuracy and fluency, facilitating deeper analysis of MT outputs. Our analysis, powered by this expert dataset, reveals that state-of-the-art MT systems may have surpassed human translations in accuracy while still lagging in fluency - a critical distinction obscured by existing accuracy-biased metrics. Our findings highlight that advancing MT evaluation requires not only better protocols but crucially, high-quality annotations from skilled linguists.