Large language models can flag risky classical translation outputs without references
Can We Triage LLM Translation Errors in Classical Texts Without Human References? Source Novelty, GEMBA Scoring, and Budgeted Review through Pali-to-English Translation
Computation and Language
Summary
When translating old texts, it’s hard to know which sentences might have mistakes, especially without correct human versions to compare. The authors tested ways to spot risky translations automatically using three language models translating Pali texts into English. They found that using a special scoring method called GEMBA, which judges translations without needing the original correct answer, best identified likely errors. Combining this with other checks can help decide which translations need human review, saving experts time.
What this means in practice
- •For language technology teams: Identify which machine-generated translations of classical texts need expert human checks without having correct reference translations.
- •For cultural heritage digitization teams: Prioritize human review of translations in large projects by using model-based scoring to focus on risky passages in classical languages like Pali.
Tested on one dataset.
Authors
Máté Metzger
Abstract
As large language models become capable translators of classical texts, a key challenge is deciding which outputs need expert review when no human reference exists. This study tests reference-free error triage through Pali-to-English translation. Three LLMs translated 15,493 passages. Five signals were compared: source novelty, source-candidate embedding distance, peer-translation disagreement, English-to-Pali backtranslation, and no-reference GEMBA scoring. Signals were calibrated on a 3,000-item reference-informed LLM-adjudicated sample and checked against a 500-item author-adjudicated anchor. Human references supported calibration and validation only; they were never used to compute the risk signals. Source novelty was a useful source-side risk prior but not a per-candidate error detector. Peer disagreement and backtranslation provided secondary signal. The strongest method was no-reference GEMBA scoring by a panel of models generally regarded as stronger than the translators: reviewing the top 10% by GEMBA risk captured 81.6% of panel-major errors in the calibration set. GEMBA also remained the best reference-free signal against the author anchor. A same-tier panel, with self-scoring excluded, remained useful but performed worse, indicating that evaluator strength matters beyond the prompt alone. A budgeted workflow is proposed, combining source novelty, peer disagreement, and stronger candidate-aware judging to allocate human review. Transfer to other classical languages, including Latin, Ancient Greek, and Sanskrit, remains to be tested.