How Far Can Machine Translation Quality Take You? Extrinsic Discourse Evaluation in Goal-Oriented Setups
2026-06-15 • Computation and Language
Computation and Language
AI summaryⓘ
The authors looked at how machine translation errors affect understanding across sentences, not just isolated translation quality. They tested this in two ways: by counting how often entities are mentioned correctly in static texts and by seeing how translation impacts teamwork in a game. They found that even good translation systems can make mistakes that mess up understanding and coordination. Their work suggests we should evaluate translations based on real-world communication tasks, not only on direct translation accuracy.
machine translationdiscourse evaluationreferential consistencyentity countinginteractive communicationmulti-agent systemscoordinationWelfare Diplomacy gameextrinsic evaluation
Authors
Wafaa Mohammed, Kata Naszadi, Vlad Niculae
Abstract
Existing machine translation (MT) metrics and discourse-focused evaluations primarily assess translation quality intrinsically, without measuring the downstream consequences of translation errors. In this work, we focus on extrinsic discourse evaluation of machine translation under two distinct regimes: static and interactive. Under the static regime, we propose an entity counting task as a probe of referential consistency in discourse. We show that high intrinsic MT quality does not reliably predict downstream discourse success and strong MT systems still produce referential inconsistencies. For the interactive regime, we study the goal-oriented multi-agent Welfare Diplomacy game as a probe of long-horizon communication and coordination. We find that interaction-specific translation failures impact downstream coordination. Our results highlight goal-oriented environments as a viable framework for discourse-sensitive extrinsic MT evaluation.