Large language models struggle with moroccan darija figurative language translations
Towards Figurative Language Understanding in Moroccan Darija: An Evaluation of Large Language Models
Computation and Language
Summary
Figurative language, like idioms and metaphors, is hard for machine translation systems because their meaning often goes beyond the words. The authors created a special dataset to test how well large language models handle figurative expressions in Moroccan Darija, a local Arabic dialect. They found that some models translate these expressions better than others, but understanding the meaning in context is still a challenge. The study also shows that generating a good translation and recognizing the right meaning are two different tasks for these models.
What this means in practice
- •For machine translation developers: Improve Moroccan Darija dialect translation by testing and refining models against the MDFC benchmark dataset for figurative language.
- •For natural language processing engineers: Use layer-wise probing insights to enhance models' ability to distinguish figurative from literal language in dialectal Arabic applications.
Authors
Jihad R'baiti, Youssef Hmamouche, Emmanuele Chersoni, Chu-Ren Huang, Amal El Fallah Seghrouchni
Abstract
Figurative language remains a major challenge for Machine Translation (MT) systems, as the intended meaning of such expressions often goes beyond literal translation. Although recent advances in Large Language Models (LLMs) improved the translation of Arabic dialects, benchmark resources for evaluating figurative language translation in Moroccan Darija remain scarce. To this end, we introduce the Moroccan Darija Figurative Language Corpus (MDFC), a benchmark dataset for MT and multiple-choice question (MCQ) selection covering idioms, metaphors, and proverbs. Using MDFC, we evaluate 19 open- and closed-weight language models and MT systems at both the expression and contextual levels under zero-shot and six-shot settings. We also conduct layer-wise similarity-based probing to determine how model representations distinguish figurative from literal meanings. Our findings reveal that closed-weight models achieve the strongest translation performance, while the benefits of contextual information vary across models and tasks. Differences between translation performance and MCQ accuracy highlight the distinction between generating an appropriate figurative translation and recognizing the intended meaning among candidate translations.