Automatic translation metrics struggle with social media culture
When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation
Computation and Language
Summary
Current automatic tools that check translation quality don’t work well for social media posts because these posts use lots of slang, jokes, and cultural references. The authors found that popular metrics often give worse translations higher scores, which is misleading. They created a new training method called CuRIL that helps a smaller AI model understand cultural context better, improving its ability to judge social media translations more like humans. Their improved system helps reduce poor translations significantly when used to guide translation improvements.
What this means in practice
- •For machine translation teams: Improve social media translation quality by using cultural reasoning–enabled evaluation metrics to detect poor translations accurately.
- •For natural language processing engineers: Train smaller language models to judge translations in culturally rich settings, reducing reliance on very large models for evaluation.
Authors
Yiwen Qiu, Linjuan Wu, Dingming Li, Yizhou Liu, Zixuan Wang, Haolei Xu, Ye Guo, Daoxin Zhang, Weiming Lu, Yongliang Shen
Abstract
Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.