Large language models produce noisy translations cleaned by new benchmark
TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs
Computation and Language
Summary
Sometimes when large language models translate text, they add extra words like labels or repeated phrases that aren’t part of the actual translation. The authors studied many examples of these noisy translations and found common patterns of this extra text. They created TransClean, a dataset to help test methods that can automatically remove the unwanted parts and keep only the clean translation. They also compared two ways to extract the clean translations: one that detects important parts using quality estimates and another that asks a language model to find the right text.
What this means in practice
- •For machine translation developers: Improve translation tools by automatically removing extra non-translation text for clearer outputs.
- •For content moderation teams: Clean multilingual text outputs from AI to ensure only intended translations are shared or published.
Authors
Shenbin Qian, Yves Scherrer
Abstract
Large language models (LLMs) are increasingly used for machine translation, yet their outputs often contain additional text beyond the translation itself, such as language labels, explanations or bilingual repetitions, which we term translation noise. Despite its prevalence, this problem lacks dedicated benchmarks and systematic study. We analyze over 790,000 translation outputs from 12 LLMs across 22 language pairs (LPs) and identify 12 recurring noise patterns, which we group into formatting and content noise. Building on the observed patterns, we construct TransClean, a controlled benchmark of 9,900 pairs of noisy and clean translation outputs, comprising 8,800 synthetically generated instances and 1,100 manually curated authentic instances. We evaluate two extraction approaches on the TransClean benchmark: 1) a span-based extraction method leveraging translation quality estimation models for span detection, and 2) an LLM-based extraction method that prompts an LLM to isolate the translation. Our benchmark and analysis provide the first systematic framework to evaluate and improve the cleanliness of LLM translation outputs.