Machine translation accuracy improves by accounting for term variation
Improving Term Evaluation in Machine Translation: Variation Matters
Computation and Language
Summary
Machine translation systems usually treat each word or term as having only one correct translation, but human translators often use different valid versions of the same term. The authors studied English-to-French scientific translations and found that current evaluation methods wrongly penalize translations for these natural variations. They introduced a new way to measure whether variations in the translation reflect variations in the source text. Their work shows that recognizing valid variations can make evaluation of translation quality more accurate and fair.
What this means in practice
- •For machine translation engineers: Enhance translation evaluation metrics by allowing valid term variation so that system quality assessments better reflect real-world human translation practice.
- •For localization teams: Use variation-aware evaluation to balance glossary constraints and natural term diversity for more fluent and accurate localized scientific content.
Authors
Nicolas Dahan, Ziqian Peng, François Yvon, Rachel Bawden
Abstract
Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.