Pterodactyl improves low-cost automated translation quality evaluation
Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation
Computation and Language
Summary
Checking the quality of translated text can be expensive and slow. The authors created Pterodactyl, a tool that uses large language models to quickly decide if a translation is good or if it has errors. Their system is very customizable and cheap to run, making it easier to sort translations and find mistakes automatically. They also made special versions of the system to better spot perfect translations without errors.
What this means in practice
- •For machine translation developers: Automatically score translation segments to improve training data quality.
- •For content localization managers: Identify error-free translated segments to accelerate publishing workflows.
Authors
Evelyn Y. Garland, Carola F. Berger
Abstract
In this paper, we present 1020 Group's submission to the WMT26 shared task on automated quality evaluation. Our system, Pterodactyl, was engineered specifically for subtasks 2 and 3, that is, segment-level quality score prediction and the detection of error-free segments. Pterodactyl is a highly customizable, ultra-low cost system optimized for binary classification in translation triage and data curation. It is based on an LLM-as-a-judge approach, with prompt engineering and model selection guided by ROC analysis. We also developed several variants of Pterodactyl that were specifically adapted for subtasks 3.