Social media translation improves with images and cultural context
Beyond Words Alone: Disentangling Distributed Evidence for Visually Grounded Cultural Translation
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Translating posts on social media is tricky because their meaning often depends not just on the words but also on pictures and culture. The authors created a special test set called Culture-MMT to see how much images and cultural clues help in translation. They found that using real images helps translate posts that rely on visual information, but less so for symbolic content. This shows that good translation needs both understanding pictures and adapting to culture.
What this means in practice
- •For social media platform developers: Improve automatic translation quality for multilingual user posts by integrating relevant images and cultural knowledge as shown by this paper’s benchmark.$Commercial implications: This paper enables development of better multimodal translation tools that platforms can offer to users for seamless cross-cultural communication.
- •For machine translation engineers: Test and refine translation models using a diagnostic dataset that distinguishes errors caused by missing images versus cultural adaptation challenges.
Authors
Dingming Li, Yiwen Qiu, Linjuan Wu, Hongxing Li, Zixuan Wang, Xinze Lyu, Ye Guo, Daoxin Zhang, Zhe Xu, Weiming Lu, Yongliang Shen
Abstract
Social media enables cross-lingual communication, yet the meaning of user-generated content (UGC) is often distributed across text, images, and cultural knowledge, making translation challenging. Existing evaluations typically treat images as optional and assume textual information is sufficient, obscuring whether errors stem from missing visual evidence or inadequate cultural adaptation. We introduce \textbf{Culture-MMT}, a diagnostic benchmark for \textbf{distributed-evidence pragmatic reconstruction} in social-media translation. It contains 498 curated, de-identified UGC instances across 20 domains, categorized into Symbol, Visual, and Hybrid by their required evidence. We also develop a human-aligned domain-specific judge, Judger, and a controlled evaluation framework that systematically varies the evidence available to translators. Across 17 model configurations, real images improve performance by +0.280 on Visual and +0.197 on Hybrid, but only +0.050 on Symbol, while random images provide no comparable gains. Human-authored descriptions further show that jointly dependent cases remain more challenging even when visual facts are explicit. These results characterize culturally effective social-media translation as {meaning reconstruction from distributed evidence}, highlighting visual grounding and cultural-pragmatic adaptation as distinct model bottlenecks.