Vision language models struggle with detailed multilingual image understanding
MultiInpaintCOCO: Fine-Grained Visual Understanding in Multiple Languages
Computation and Language
Summary
Some computer programs that look at pictures and read descriptions in many languages have a hard time noticing small details in images. The authors created a new test set in 13 different languages to study how well these programs understand fine image details in languages other than English. They found that the programs do worse when the language has fewer resources, and they often prefer English captions even if they are incorrect. This bias depends on the program's design. The authors shared their new dataset and results online.
What this means in practice
- •For multilingual ai developers: Test and improve image understanding models across multiple languages by using the MultiInpaintCOCO dataset to evaluate fine-grained visual comprehension and language bias.
- •For multilingual content moderation teams: Detect and reduce language bias in automated image captioning or retrieval systems to ensure fair performance for diverse language users.
Authors
Andrei-Alexandru Manea, Jindřich Libovický, Philipp J. Rösch
Abstract
Previous research shows that vision-language (VL) models struggle to understand fine-grained image details and tend to focus on the overall layout. Moreover, any weakness in English tends to be further amplified in the multilingual setting. To study that, we extend InpaintCOCO, an English dataset focused on fine-grained visual details, to 13 typologically diverse languages across five language families, creating MultiInpaintCOCO - a new multilingual benchmark of 953 minimal-pair examples. We further test several state-of-the-art models in multilingual multimodal retrieval, considering both light VL CLIP-like encoders and large language models. Our results show that performance degrades substantially for lower-resource languages, with the strongest predictor being the data resources available for the languages rather than typological similarity to English. In cross-lingual retrieval, most models exhibit a systematic bias toward English: when given a correct caption in a non-English language and an incorrect one in English, models preferentially select the English caption. This bias varies considerably across architectures. Among the tested models, Gemma 3 shows the strongest English preference while Qwen 3 is the least biased, despite both reaching similar monolingual accuracy across languages. The code is available at https://github.com/ufal/multi-inpaint-coco.