Tactics improves machine translation tests by smartly picking diverse text
TACTICS: Taxonomy-Aware Intelligent Corpus Sampling for Machine Translation
Computation and LanguageMachine Learning
Summary
Machine translation systems are usually checked by testing on random samples of text, which may miss important types of language use. The authors created TACTICS, a method that organizes text samples based on detailed language categories from style guides, then selects examples that cover rare and important language features. This approach helps test machine translation more fairly and efficiently, giving clearer results with fewer examples. It also avoids false signals when systems do not actually differ in quality.
What this means in practice
- •For machine translation engineers: Improve evaluation sets by including diverse and rare language phenomena to better detect differences between translation systems.
- •For natural language processing teams: Create more representative test samples for language models by targeting underrepresented text categories derived from style guides.
Authors
Prasanth Bathala, Anubhav Shrimal, Sukhdeep Singh Kharbhanda, Pradyumna Lanka, Rohit Dhaipule
Abstract
Large-scale machine-translation (MT) systems are typically evaluated on random samples from a corpus whose distributional composition is an artifact of how it was assembled. Such a sample inherits the phenomena the collection happens to contain rather than the full space a system must handle, spanning rule-governed conventions (terminology, punctuation, currency formatting) and context-dependent phenomena (tone, honorifics, document-level coherence), and thus provides no coverage guarantee for assessing robustness. We propose TACTICS (Taxonomy-Aware Coverage-opTimized Intelligent Corpus Sampling), which recasts coverage as an explicit objective. TACTICS induces a hierarchical taxonomy from a locale style guide, classifies segments against it, and selects a fixed-budget subset jointly optimizing coverage of rare categories, document-level coherence, and distributional fidelity to the full corpus. Applied to MT evaluation across four translation directions, TACTICS improves coverage of rare categories over lexical and embedding-based selection. By targeting the phenomena that separate systems, TACTICS makes a fixed evaluation budget go further, recovering the true system ranking from far fewer segments than random sampling wherever a real quality gap exists and never signaling a difference where none exists.