Machine translation evaluation expands beyond word accuracy to real conversation success
Evaluating Communicative Success in Machine-Translated Conversation
Computation and Language
Summary
Interpreters that use machine translation help people who speak different languages talk to each other live, but usual tests only check if single sentences translate correctly. The authors created a new way to check if conversations succeed in meaning, intent, and cultural fit, not just in accurate words. Their method works for one message or a full back-and-forth conversation and was tested with several languages and setups. They found that while translation often preserves meaning, it can fail to capture intent and social cues, which current tests miss. This new approach helps measure how well machine translation supports real communication between people.
What this means in practice
- •For machine translation developers: Assess interpreter agents by measuring not just word accuracy but also meaning, intent, and cultural fit in conversation, improving translation tools for live communication.
- •For customer service teams: Enhance live multilingual chat by using evaluation tools that detect when translation fails to convey proper intent or social cues, improving service quality.
Authors
Faiz Ghifari Haznitrama, Alice Oh
Abstract
Interpreter agents built on machine translation (MT) increasingly mediate live conversation between people who do not share a language, yet we still evaluate them with metrics built for isolated sentences, which measure fidelity rather than whether communication succeeds. We introduce a reusable three-layer checklist-and-judge framework that evaluates interpreter-mediated conversation across semantic, pragmatic, and cultural-social dimensions, covering the naturalness, intent, and social appropriateness that fidelity metrics leave unmeasured. It runs in both single-turn and interactive multi-turn settings, where simulated users reply to translated messages as the conversation unfolds and each turn is scored alongside the conversation as a whole. We extensively validate it through controlled perturbations, cross-judge comparisons, and human annotations. Our main single-turn benchmark evaluates 10 interpreter setups across Arabic, Bengali, Indonesian, and Korean from 5,624 OpenSubtitles-derived scenarios spanning 12 translation directions, and our multi-turn study covers all 6 language pairs in scripted and live modes. Results show a consistent decline from semantic to pragmatic and cultural-social success, while conventional MT metrics overlook failures among stronger interpreters, and prompt ablations show that scenario context, structured instructions, and cultural context improve communicative success, although gains vary across setups. Our work thus provides an evaluation framework and benchmark for interpreter agents in conversation, and highlights the importance of communicative success alongside existing translation metrics.