ImgMT dataset benchmarks text translation inside images across 129 languages
ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation
Computation and Language
Summary
Translating text embedded in images like diagrams or maps is tricky because you have to find the text, read it, and then translate it. The authors created ImgMT, a large dataset of images with text in 129 languages from Wikipedia. They also set up ways to measure how well systems detect, read, and translate this text back into images. Their tests show that the language and script used affect translation success, and they found that images don’t add much extra help in understanding the text for translation.
What this means in practice
- •For machine translation developers: Use ImgMT to evaluate and improve systems that translate text embedded within images across many languages and writing systems.
- •For visual content localizers: Employ the ImgMT dataset for benchmarking tools that translate and reinsert text in infographics, diagrams, or maps for multilingual audiences.
Authors
Šárka Uramová, Petar Kirilov Petrov, Josef Jon, Ondřej Bojar, Michal Novák
Abstract
Text-in-image translation aims to translate all textual content embedded in an image. In Text Image Translation (TIT), the output is translated text; In-Image Machine Translation (IIMT) additionally renders the translation back into the image. We introduce ImgMT, a massively multilingual dataset and benchmark mined from localized SVG illustrations on Wikimedia Commons, focusing on diagrams, maps, and infographics. ImgMT contains 35,612 aligned image pairs grouped into 1,855 image sets, spanning 129 languages, 20+ writing systems, and 4,886 language pairs. We propose a multi-level evaluation protocol that measures source-side text-region detection and OCR accuracy alongside target-side translation quality. We benchmark several multimodal large language models and a dedicated TIT system in English->X and X->English settings, and find that the source language and writing system substantially affect TIT quality. Finally, we show that ImgMT is not well suited for isolating the contribution of visual information in multimodal translation.