Arabic dialects affect math problem solving in language models
Benchmarking Arabic Dialects for Math Problems
Computation and LanguageArtificial Intelligence
Summary
Understanding math problems in different Arabic dialects is hard for computer programs. The authors made a new set of math problems in five Arabic dialects plus standard Arabic to test how well these programs can solve math questions. They found that programs do worse when the problems include local cultural details rather than just translated words. Some programs react differently to these changes, and the way they think through the problems changes even if their overall score stays the same. This shows that both language type and culture impact how well AI can solve math in Arabic.
What this means in practice
- •For natural language processing developers: Develop and evaluate math-solving AI that better handles different Arabic dialects and cultural contexts.
- •For language model trainers: Improve training data coverage by including dialectal and culturally localized Arabic math problems to enhance reasoning performance.
- •For educational technology providers: Create AI tutors that adapt to regional Arabic dialects and local references for better student engagement in math problem solving.$Commercial implications: Enables regionally customized AI math tutoring products for Arabic-speaking markets by bridging dialect and cultural gaps.
Authors
Noor Abo Mokh, Teresa Lynn, Nizar Habash, Ahmed Elshabrawy, Sara Alhajeri, Alaa Elsetohy, Preslav Nakov, Alham Fikri Aji
Abstract
Mathematical reasoning remains underexplored for low-resource languages, including Dialectal Arabic. To bridge this gap, we introduce the DAR-MGSM (Dialectal Arabic Multilingual Grade School Math) Benchmark, the first manually curated benchmark for evaluating mathematical reasoning in Dialectal Arabic. DAR-MGSM consists of math word problems (MWPs) requiring multi-step reasoning and covers five major Arabic dialects alongside Modern Standard Arabic (MSA). The benchmark comprises two parallel versions: translated-only and translated & culturally localized. The latter version adapts entities and references to regional varieties. We evaluate eleven Arabic-centric and multilingual large language models on DAR-MGSM. The results show that the performance generally declines on the localized version, highlighting the challenges introduced by culturally grounded adaptations. However, the magnitude of this effect varies across models, with Arabic-centric models exhibiting different levels of sensitivity to localization. A mechanistic interpretability analysis shows that translated and localized variants of the same MWPs often induce distinct internal reasoning trajectories, even when aggregate accuracy remains similar. Overall, our results show that mathematical reasoning performance depends on both linguistic variety and cultural context.