Papers for

african tech product teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Models struggle to know nigeria facts even in english

NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria

Abstract: Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \yoruba, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\% for the US/UK but 20\% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to \yoruba{} (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66--79\% on the UK and the US but none exceed 31\% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\% on culture and 98\% on politics against 26\% on statistics, so the gap is not specific to Nigeria.

Wed 4 NovComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Most language tests measure how well models understand the language, not whether they know true facts about Africa. The authors created NaijaFacts, a test with true facts about Nigeria in several local languages and English. They found that models often do not know Nigerian facts, even when asked in English. This lack of knowledge is much bigger than any drop caused by language differences. The problem is common for many African countries, not just Nigeria.
Open → 2611.06504v1