Language models struggle to grasp igbo culture despite fluency
Do Language Models Understand Igbo Culture? Beyond Multiple Choice
Computation and Language
Summary
Language models often fail to truly understand the culture of Igbo-speaking people, even when they can speak the language well. The authors created a new test based on authentic Igbo culture, checked by native speakers, to better evaluate this. They found that many models do badly on cultural knowledge but can still answer questions correctly by guessing. Only one model showed improved cultural understanding when using Igbo. The study highlights that true cultural understanding needs human evaluation and that simple tests can be misleading.
What this means in practice
- •For language technology developers: Evaluate language model cultural understanding more accurately using indigenous benchmarks and human evaluation to avoid misleading results from multiple-choice tests.
- •For natural language processing teams: Improve language models’ handling of regional and cultural nuances by incorporating native language prompting and persona information.
Authors
Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad
Abstract
The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use the Igbo language as a case study. Existing benchmarks test translated Western ideas, lacking depth. To address this, we introduce CAB-Igbo, a benchmark developed from indigenous cultural sources and verified by native speakers across six domains. We evaluated diverse models under varying conditions, incorporating regional personas. Performance was assessed using multiple-choice accuracy and human evaluations across Epistemic Fidelity, Representational Richness, Linguistic Quality, and Pragmatic Proficiency. Results reveal a stark dissociation between language fluency and cultural knowledge. Through statistical testing, we show that for most models, the severe lack of Epistemic Fidelity and Representational Richness is already present in English and does not significantly worsen when switching to Igbo. Instead, native language prompting mainly degrades Linguistic Quality and Pragmatic Proficiency. Gemini 2.0 Flash is the only model whose cultural knowledge scores improve under Igbo prompting. We find that current models struggle to demonstrate the knowledge needed for true alignment with Igbo, and multiple-choice formats allow pattern-matching to appear as understanding. Furthermore, automated judges agree with humans on linguistic quality but fail on epistemic fidelity. Ultimately, cultural evaluation must be built from community knowledge, establishing human evaluation as the gold standard to prevent misrepresenting low-resource communities globally.