Metric based loss weighting boosts image guided language translation accuracy
Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting
Computation and LanguageArtificial Intelligence
Summary
Sometimes translating text is tricky because the words can be unclear without other clues like pictures. The authors found a way to teach language models to pay more attention to images when they help make translation clearer. They measure where images really matter by comparing predictions with and without the picture, then focus more on those tricky words during learning. This new approach makes translations that use images more accurate without losing overall quality.
What this means in practice
- •For machine translation developers: Improve translation systems by training models to better incorporate image context for ambiguous words, increasing accuracy in image-guided language translation.
- •For software engineers creating assistive technologies: Enhance assistive translation tools by integrating visual context sensitivity, enabling clearer communication in multilingual environments with related imagery.
Authors
Paweł Mąka, Piotr Andruszkiewicz, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis
Abstract
Multimodal Machine Translation aims to incorporate additional signal from non-textual modalities to improve translations by resolving ambiguities. While models, through multimodal fusion, are able to accept images related to the source text, they can ignore this information. Therefore, increasing their visual sensitivity remains an active research area. In this work, we introduce a training method, Metric-based Loss Weighting, that improves visual grounding of translations by increasing the loss function for tokens that benefit from the accompanying image. We identify these tokens using the Point-wise Cross-mutual Information (PCXMI) metric, which compares the model's output probabilities with and without visual context. We introduce a Congruency-based PCXMI metric and experimentally show that both metrics working in combination yield the best results. We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models on the task of Image-guided Machine Translation for three language directions. Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset, improving accuracy by up to more than 7 percentage points compared to standard fine-tuning, while maintaining strong general translation performance.