Machine translation meta-evaluation balances adequacy and fluency tradeoffs
Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translation
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Evaluating machine translation quality is tricky because it involves balancing how accurate the translation is (adequacy) and how natural it sounds (fluency). The authors found that this balance changes depending on the set of translation systems tested, which isn't stable over time or languages. They developed a method to adjust the importance of adequacy versus fluency in evaluation without distorting the set of real translation systems. Their approach also keeps evaluations internally consistent and works better than previous methods. This helps make translation quality checks more reliable, especially in critical areas.
What this means in practice
- •For machine translation engineers: Control the balance between accuracy and naturalness when evaluating translation models to improve metric reliability in diverse datasets.
- •For natural language processing platform teams: Tune translation quality assessments to better fit sensitive use cases by adjusting adequacy-fluency weighting while preserving evaluation consistency.
Authors
Behzad Shayegh, Niloofar Kazemi
Abstract
There is a tradeoff in machine translation meta-evaluation between prioritizing alignment with adequacy versus fluency. The balance depends on the combination of translation systems in the meta-evaluation dataset. This system set is a small, filtered sample whose characteristics change heavily across years and language pairs; it does not represent the true system distribution. Consequently, the adequacy-fluency balance is often unrepresentative and subject to change. For sensitive domains, controlling this balance is critical. We expose this balance as a tunable choice. To achieve a target balance, we reweight existing systems while minimizing distortion from uniform weighting, ensuring the evaluated systems remain real and representative. We provide an exact optimization algorithm with theoretical guarantees and pruning mechanisms to compute these weights. To validate meta-evaluation internal consistency, we design a scorer-augmentation framework that establishes a known relative identity for the scorers. Results demonstrate that our reweighting method effectively controls the adequacy-fluency balance and preserves the internal consistency, outperforming prior approaches. Finally, we analyze the performance of popular scorers across a sweep of this parameter.