betaFeedback

WorksComputer Science

NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria

Abstract

Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce NaijaFacts, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, Yorùbá, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49-50% for the US/UK but 20% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to Yorùbá (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66-79% on the UK and the US but none exceed 31% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96% on culture and 98% on politics against 26% on statistics, so the gap is not specific to Nigeria.

Introduction

Large language models (LLMs) are increasingly used as sources of factual information, yet their factual reliability is uneven and hard to audit [24, 29]. The problem compounds for low-resource languages and for facts about under-represented regions. The languages spoken across Nigeria illustrate the gap. As Figure 1 shows, they are spoken by tens to hundreds of millions of people, yet occupy a small share of the digital text on which models are pretrained. A model can therefore fail on a Nigerian fact for two different reasons: because it cannot process the language, or because the fact was never well represented in its training data. Existing multilingual benchmarks for African languages, such as AfriQA [22], AfroBench [23], and IrokoBench/AfriMMLU [1], largely evaluate models by translating generic questions into African languages, which does not isolate whether a model holds grounded knowledge about Africa itself.

Scatter plot of languages by web representation and number of speakers

Figure 1. The NaijaFacts languages are widely spoken yet barely present in web text. Each point is a language, placed by Wikipedia articles [30] and speakers [7], both on log scales. The green languages have 1–3 orders of magnitude fewer articles than high-resource ones with as many speakers.

We argue these two failure modes should be measured separately. Consider the question, what was Nigeria’s total fertility rate in the 2023? A model can fail this in two different ways: it knows the value in English but produces a wrong answer when asked in Yorùbá (a language effect), or it never encoded the value and is wrong in every language (a knowledge effect).

We introduce NaijaFacts, a benchmark of 100 factual questions about national demographic and health statistics, authored by native speakers of the Nigerian languages and translated to English. Its ground truth is authoritative, quantitative official statistics, primarily demographic and health indicators from bodies such as the National Population Commission (NPC), the National Bureau of Statistics (NBS), and the Nigeria Demographic and Health Survey (NDHS). Each question is available in English, Nigerian Pidgin, Yorùbá, Igbo, and Hausa, and is asked about three countries, giving 1,500 graded items per model. By holding the fact fixed and varying only the language of the query, NaijaFacts separates the language and knowledge effects.

A confound remains: a Nigeria-versus-English gap could reflect the resource level of the query language instead of the model’s knowledge of the country. We therefore also vary the country a question is asked about, answering the same questions for the United Kingdom and the United States against each country’s own official statistics. Crossing the five query languages with the three countries gives a country × language matrix in which the language effect is read down a column and country familiarity across a row.

The same construction extends beyond Nigeria. Reusing the pipeline, we build verified English question sets of comparable size for each of the 54 African countries, drawing every answer from that country’s own official statistics, so that we can test whether the findings hold continent-wide.

We make three contributions. First, we build a benchmark that pairs official-statistic ground truth with five-language coverage and a country × language design, which separates the language and knowledge failure modes. Second, we show that the knowledge effect dominates: eight models, spanning three geopolitical origins and including two Nigeria-adapted 8B open-weight models, hold UK and US statistics far better than Nigerian ones, and answering in English recovers almost nothing. Third, we extend the construction to all 54 African countries, which shows where the deficit generalises and which facts escape it.

Related Work

Work outside the Africa-focused research has varied the query language and the subject, but separately. Varying the query language, X-FACTR [11] and mLAMA [13] probe entity-level facts with cloze prompts, Qi et al. [25] show that scale improves per-language accuracy but not cross-language consistency, and Aggarwal et al. [2] find on 10,000 country-related facts in 13 languages that a model can recall a fact in one language and fail on it in another. Varying the subject instead, WorldBench [16] asks the same World Bank indicator questions about different countries and finds sizeable geographic disparities, while PopQA [15] and Head-to-Tail [26] show recall drops sharply for entities that are less popular online.

Closest to our domain split, LocalBench [9] reports 56.8% on narrative questions about US counties against under 15.5% on numerical ones, the same asymmetry we measure across Africa (§ 6.3). None of these crosses country with language, and most use knowledge-base triples or indicator databases as ground truth. Two further distinctions we borrow: Calderon et al. [6] separate unencoded facts from encoded but unretrieved ones, which our answer-in-English ablation tests (§ 6.1), and SimpleQA [29] measures atomic-fact correctness.

Benchmarks for African languages cover our languages but mostly not our kind of fact. AfriQA [22], Belebele [4], IrokoBench/AfriMMLU [1], and AfroBench [23] evaluate at increasing scale on translated or generic content, testing whether a model can operate in a language and not whether it knows facts about the region where the language is spoken. Uhura [5] moves toward knowledge with human-translated science and truthfulness questions in six African languages, though its content is still translated from English. Two probes are regionally grounded: GeoMLAMA [32] reports that a country’s native language is not always the best probe of knowledge about it, and BLEnD [17] covers everyday cultural knowledge in 13 languages including Hausa.

Afri-MCQA [27], which is similar to our aim collects 7.5k culturally grounded question-answer pairs written by native speakers across 15 African languages and, like us, sets out to separate linguistic competence from knowledge, using external control tasks (AfriXNLI, AfriMMLU) as the language baseline. It concludes that language understanding is the dominant limitation, whereas we find the knowledge gap dominates. The difference is instructive: their ground truth is everyday culture, which models largely know, whereas ours is the quantitative official record, which they largely do not. NaijaFacts therefore shares its goal and WorldBench’s country axis, and adds two things neither provides. Its ground truth is the quantitative record of a country’s own statistical office, and it varies the country asked about while holding the question and its five language renderings fixed, with a second foreign control that measures raw online prominence on its own. The resource skew behind the whole gap [12, 14] is what Figure 1 depicts and NaijaFacts measures. The nearest neighbours each supply some of these properties but not the combination (Table 7, Appendix § 13).

Benchmark# Afr. lang.Facts about Africa?Ground-truth sourceCountry ×\times lang.?Dominant task(s)
AfriQA [22]10partialWikipedia / web×\timesCross-lingual QA
Belebele [4]29×\timestranslated passages×\timesReading comprehension
IrokoBench [1]17×\timestranslated (MMLU/MGSM/NLI)×\timesMMLU, Math, NLI
AfroBench [23]64×\timesaggregated / translated×\times15 NLP tasks
Uhura [5]6×\timestranslated ARC / TruthfulQA×\timesScience QA, truthfulness
Afri-MCQA [27]15✓\checkmark (culture)native-authored culture QA×\timesMultimodal cultural QA
GeoMLAMA [32]1 (Swahili)partialtemplated commonsensepartialCommonsense probing
BLEnD [17]1 (Hausa)partialhand-crafted culture QA×\timesEveryday knowledge
WorldBench [16]×\timespartialWorld Bank indicators×\times (country only)Country factual recall
LocalBench [9]×\times×\timesUS Census, news, Reddit×\times (county only)Local knowledge, reasoning
NaijaFacts (Ours)5✓\checkmark (Nigeria)official statistics (NPC, NBS, NDHS)✓\checkmark (NG/UK/US)Grounded factual / stats QA

Table 7. NaijaFacts in context. Unlike prior work, NaijaFacts grounds every item in authoritative national statistics about Nigeria and crosses the country asked about with the language asked in. GeoMLAMA and BLEnD are the closest geo/cultural probes but use commonsense or everyday-culture ground truth and do not provide a matched country by language control. Afri-MCQA shares our aim of separating language from knowledge but grounds answers in culture; LocalBench grounds answers in official statistics but for US counties in English only. Belebele reports 122 language variants in total but no African subtotal; the 29 given here is our own count of its variants indigenous to or principally spoken in Africa, which includes Afrikaans and the Moroccan and Egyptian Arabic variants.

The NaijaFacts Benchmark

NaijaFacts puts 1,500 graded items to every model: 100 factual questions about national demographic and health statistics, each posed in five languages and asked about three countries, Nigeria, the United Kingdom and the United States. The questions are authored in the Nigerian languages, not translated into them, and every answer comes from authoritative official statistics. Because each country is asked the same question, a difference between countries reflects the answer, not question difficulty or phrasing. That is what separates a language effect from a knowledge effect.

Questions

The benchmark begins from 100 questions written by native speakers and statistics experts of Yorùbá, Igbo, Hausa, and Nigerian Pidgin from Obafemi Awolowo University. Authoring natively in each language keeps the questions fluent and culturally natural, and avoids translationese in the low-resource languages. They target demographic and health indicators such as population counts, fertility rates, census methodology, and survey findings. The same team then translates each question into English, and we review it, giving five renderings per question. We lightly edit questions so each is self-contained and answerable for a given country.

Ground truth

For every question we collect a gold answer for each of the three countries. For Nigeria, the native speakers who authored the questions also provide the answers in the Nigerian languages, grounded in the official national statistics of the National Population Commission (NPC), National Bureau of Statistics (NBS), and Nigeria Demographic and Health Survey (NDHS). For the United Kingdom and the United States, where these speakers are not the domain authority, we obtain answers differently. A deep-research assistant retrieves the relevant figure from official statistics, the Office for National Statistics for the UK and the U.S. Census Bureau for the US. The four native speakers who authored the questions then verify and correct each one. Grounding answers in official statistics gives unambiguous, citable ground truth, and for the UK/US answers retrieving the figure before writing it minimises fabrication. Every item records the body, release and reference year behind its answer, and those years span 2006 to 2025. Table 8 breaks the items down by source.

CountrySource bodyItems
NigeriaNigeria Demographic and Health Survey 2024 [8]81
NigeriaNPC census documentation, 2006 and 2023 [20]8
NigeriaNBS releases carrying no edition year [18]11
UKOffice for National Statistics [21]56
UKUK government departments11
UKNational Records of Scotland and Scotland’s Census [21]9
UKWHO/UNICEF Joint Monitoring Programme [31]8
UKUK legislation5
UKother official sources11
USNational Center for Health Statistics [19]42
USU.S. Census Bureau [28]38
USWHO/UNICEF Joint Monitoring Programme [31]9
USU.S. Environmental Protection Agency4
USother official sources7

Table 8. Where the gold answers come from, by country. Reference years span 2006 to 2025 and are recorded in the bibliography entry for each body. Every UK and US item additionally stores the URL of the specific release it was taken from, and those URLs are distributed with the benchmark; the Nigerian items name their census or survey edition in the question text.

Coverage and item contents

Each question is instantiated for all three countries by substituting the country it refers to, using a country-specific analog where a direct substitution would be unnatural, and this is done in every language. The five language renderings therefore exist for Nigeria, the UK and the US alike, so all fifteen country by language cells hold real items rather than gaps. Each question is stored with its text in each language, the gold answer for each country, the kind of answer it expects (a number, a named entity, a date, or other), and the provenance of each answer. Table 4 summarises the dataset (Appendix § 11) and Appendix § 10 gives an example item in all five languages.

ComponentValue
Questions100, authored by native speakers
Question languagesYorùbá, Igbo, Hausa, Nigerian Pidgin (native); English (translated)
Answer countriesNigeria (NPC/NBS/NDHS), UK (ONS), US (Census Bureau/NCHS)
Answer collectionNigeria: native speakers, in-language; UK/US: deep-research, verified
Evaluation cells100×5100 \times5 languages ×3\times3 countries =1,500= 1{,}500 per model
Cross-country extension54 African countries, 118–125 English items each (6,689 total)

Table 4. NaijaFacts at a glance. The same questions are answered for three countries, so the country comparison is matched by construction; the 54-country extension instead matches a fixed blueprint (domain mix, answer-type distribution) with each country’s facts drawn from its own official sources.

Two properties qualify what the set measures. First, the statistics cluster by release. A model that has memorised one table can therefore answer many items at once: 81 Nigerian items come from one survey report, and 26 US items from one NCHS report. There are fewer independent facts here than there are items. Second, three items ask for explanations rather than values, such as why a post-enumeration survey follows a census. We keep them so that every item is scored in all five languages and three countries, and we grade them more leniently.

Extension to 54 countries

To test whether the Nigeria results are idiosyncratic, we reuse the same recipe to build an English-only set for every African country. Each country gets 118–125 verified items built to a fixed blueprint, which holds the domain mix and answer-type distribution constant so that countries stay comparable. Every answer comes from that country’s own record. Statistical values are fetched programmatically from the World Bank World Development Indicators and the DHS Program APIs, never transcribed by a model, so a numeric gold answer cannot be hallucinated and verifying one is simply a re-fetch. Items whose value is missing or ambiguous at fetch time are dropped, which is why the per-country count varies. Because instruments genuinely disagree, each indicator is pinned to a single instrument across all countries. This yields 6,689 items spanning statistics alongside smaller culture and politics subsets, which lets us ask not just whether the deficit generalises but which kinds of fact it spares (§ 6.3); full provenance is in Appendix § 14.

Experimental Setup

We evaluate eight models (Table 5, Appendix § 12): six frontier models spanning three geopolitical origins (US, EU, CN), and two Nigeria-adapted open-weight models, N-ATLaS and AfroLlama-V1, both 8B Llama-3 fine-tunes that test whether targeted language adaptation helps. Frontier models are accessed through AWS Bedrock; the 8B models run locally on a single 24 GB GPU. All models share one harness and identical prompts, with a 4,096-token answer budget. Sampling is left at each provider’s default for the hosted models, because Claude Opus 5 rejects an explicit temperature or top-pp; the two local open-weight models are sampled at temperature 0.1.

ModelOriginIdentifierSize
Frontier (proprietary, via AWS Bedrock)
Claude Opus 5USanthropic.claude-opus-5–
GPT-5.6USopenai.gpt-5.6-sol–
Grok 4.6USxai.grok-4.6–
Mistral Large 3EUmistral.mistral-large-3-675b-instruct675B
Qwen3CNqwen.qwen3-235b-a22b-2507235B
Kimi K2.5CNmoonshotai.kimi-k2.5–
Nigeria-adapted (open weight, Llama-3-8B fine-tunes, local)
N-ATLaSNGNCAIR1/N-ATLaS8B
AfroLlama-V1NGJacaranda/AfroLlama_V18B

Table 5. Models evaluated. The frontier set spans three geopolitical origins (US, EU, CN), enabling the question of whether a model’s origin affects whose facts it encodes; the Nigeria-adapted models are 8B Llama-3 fine-tunes targeting Nigerian languages. Proprietary parameter counts are undisclosed. Cross-region inference-profile prefixes are omitted from the identifiers.

The mainline condition is native: each question is asked in the target language and answered in that language, as a speaker would actually use the model; the system prompt requests only the answer, with no explanation (exact prompts in Appendix § 8). An answer-in-English analysis condition (§ 6.1) keeps the question in the target language but requests the answer in English, testing whether failures are expression failures. For the country analysis, each question explicitly names the country it concerns (with a country-specific analog where a direct swap would be unnatural) and is scored against that country’s gold answer, yielding the country ×\times language matrix of § 5.

Answers are free text, so we score with light normalisation and then an LLM-as-judge (Claude Opus 5) that returns a binary verdict. It accepts formatting, punctuation and rounding variants of the correct value, and marks materially different figures, wrong entities, refusals and empty responses incorrect. On the cross-country sets the judge additionally receives per-item grading notes listing accepted variants. The full judge prompt is in Appendix § 9; unresolved verdicts are marked unclear and excluded (9 of 16,000 judgments, under 0.1%). Judged accuracy, broken down by language and country, is the primary metric. We leave human annotators and a second judge to future work.

ModelSizeEnglishPidginHausaIgboYorùbáAvg. (NG lang)
GPT-5.6–282629312227.0
Claude Opus 5–282621252424.0
Grok 4.6–292325251923.0
Kimi K2.5–1517139511.0
Qwen3235B18168859.2
Mistral Large 3675B17147226.2
N-ATLaS8B151112689.2
AfroLlama-V18B1176054.5

Table 1. Main result, the language effect. Judged accuracy on the Nigeria statistics set, 100 items per language, native condition. Frontier models degrade from English in the order Pidgin > Hausa ≈ Igbo > Yorùbá, tracking web representation. The adapted 8B models (bottom block) are far weaker and degrade in the same direction, so language adaptation does not by itself supply Nigerian facts.

Results

Language effect vs. knowledge effect

Table 1 reports judged accuracy on the Nigeria statistics set by query language in the native condition, with a per-model bar chart in Figure 5 (Appendix § 15). The language effect is real and tracks resource level. Averaged over the eight models, accuracy falls monotonically from English to Yorùbá in the order given by web representation (Figure 1): Pidgin, an English-derived creole, costs under 3 points, while Yorùbá, the least web-represented of the four, costs 9. The effect is far larger for some models: Claude Opus 5, GPT-5.6, and Grok 4.6 lose 14–35% of their English accuracy in Yorùbá, whereas Qwen3, Kimi K2.5, and Mistral Large 3 lose 67–88%, collapsing to 2–5% absolute.

Bar chart showing judged accuracy on the Nigeria statistics set by query language

Figure 5. Judged accuracy on the Nigeria statistics set by query language (native condition), one bar per model within each language cluster; languages ordered by web representation and Nigeria-adapted models hatched. Frontier models degrade as representation falls, while the adapted 8B models stay flat across languages at a lower level.

The knowledge effect dominates the language effect. Even in English, the best model reaches only 29% on Nigerian statistics, while the same three strongest models score 66–79% on the matched US and UK sets. Averaged over models, the English-language gap between the US/UK and Nigeria is 30 points (Table 6), more than three times the largest language effect on the Nigeria set. Most failures on Nigerian facts are therefore not translation failures: the fact is absent in every language. This inverts the ordering Tonja et al. [27] report for African cultural knowledge, where the language effect is the larger of the two, and the domain is what differs. Everyday culture is well represented in web text, so what remains to be lost there is expression, while the statistical record is largely absent, so language has little left to subtract.

Language asked inNigeriaUKUS
English20.149.250.2
Nigerian lang. (avg)14.335.534.9
Pidgin17.544.944.2
Hausa15.135.235.1
Igbo13.234.233.5
Yorùbá11.227.526.6

Table 6. Country by language matrix, judged accuracy (%) averaged over all eight models (native condition; 100 items per language per country per model). Reading across a row isolates country familiarity at fixed language: the Nigeria deficit is ∼\sim30 points in English and persists in every language. Reading down a column isolates the language effect at fixed country: the drop from English to Yorùbá is 9 points on Nigeria but 22–24 points on the UK/US, where models actually hold the knowledge. The near-identical UK and US columns bound the pure prominence effect at ≲\lesssim1 point.

Adaptation to Nigerian languages does not supply Nigerian knowledge. The Nigeria-adapted 8B models score 6–15% (N-ATLaS) and 0–11% (AfroLlama-V1), which is unsurprising at 30 to 100 times fewer parameters. They also degrade across languages in the same direction and by much the same proportion: the mean relative drop from English to Yorùbá is 51% for them against 50% for the frontier models. N-ATLaS is the more stable of the two: its coefficient of variation across the five languages is 34% against AfroLlama-V1’s 68%. The most stable models overall are the strongest frontier ones (Claude Opus 5 10%, GPT-5.6 13%, Grok 4.6 15%). Both adapted models are weakest in Igbo, but only one of them lacks Igbo adaptation. AfroLlama-V1 was never trained for it, its card listing Swahili, Xhosa, Zulu, Yorùbá, Hausa and English [10], so its 0% there is unsurprising. N-ATLaS was: its card reports roughly 200,000 Igbo instruction samples, the same budget as Hausa and Yorùbá, and its developers’ own human evaluation rates its Igbo output above its Yorùbá [3]. Yet its factual accuracy is lowest in Igbo at 6% and twice that in Hausa at 12%, so equal adaptation data and better-rated fluency did not translate into better factual recall. Fine-tuning an 8B model on Nigerian languages therefore leaves both gaps open at this scale: it neither supplies the factual record nor buys robustness to the query language. Closing the factual gap appears to need frontier scale or explicit injection of African knowledge during training.

The deficit does not depend on where a model was built. Origin does separate the models on Nigerian statistics: asked in English, the US-built systems average 28.3% against 17.0% for the European and 16.5% for the Chinese ones. But that ordering is the same on the UK and US sets, so it tracks capability rather than whose facts a model holds. The test that would show provenance is a home-country advantage, and there is none: the three US-built models score +1, −8 and +7 points on the US set relative to the UK set. No model of any origin exceeds 31% on Nigerian statistics. The converse test is unavailable, since the benchmark has no French or Chinese question set.

Country × language

Figure 2 shows accuracy as a function of the country asked about and the language asked in, with the underlying numbers in Table 6 (Appendix H).

Country × language judged accuracy heatmap

Figure 2. Country × language judged accuracy, averaged over all eight models (native condition). Columns (country asked about) separate far more than rows (language asked in): the knowledge effect dominates. The worst cell is Yorùbá × Nigeria, where both effects compound.

The expected ordering of the four conditions holds, model-averaged: A(en,US/UK)≥A(NG lang,US/UK)≥A(en,Nigeria)≥A(NG lang,Nigeria)A(\mathrm{en}, \mathrm{US/UK}) \ge A(\mathrm{NG\ lang}, \mathrm{US/UK}) \ge A(\mathrm{en}, \mathrm{Nigeria}) \ge A(\mathrm{NG\ lang}, \mathrm{Nigeria}). Since Yorùbá and Nigeria co-occur throughout the training text, asking about Nigeria in Yorùbá might have been the pairing models handle best. It is their weakest instead, at 11.2%.

Read across rows, at fixed language, the Nigeria deficit is present in all five languages but its size varies with the language: 30 points in English and 27 in Pidgin, narrowing to 20–21 in Hausa and Igbo and 16 in Yorùbá. Read down columns, at fixed country, the English-to-Yorùbá drop is 9 points on Nigeria but 22–24 on the UK/US. Both directions point the same way. Language costs most where the model has knowledge to lose, and the country deficit shrinks in the languages where models are already wrong in English.

Pure country prominence, by contrast, is near zero. The US-versus-UK gap at fixed language, our control for raw online prominence, is at most 1 point in any language (50.2 vs 49.2 in English, 26.6 vs 27.5 in Yorùbá). The US is far more present online than the UK, yet the models score them the same, so online prominence cannot be what produces the 30-point Nigeria deficit. What separates Nigeria from these two is how well the models know the country.

Analysis and Ablations

Answering in English does not recover accuracy

If a model knows a fact but cannot express it in Yorùbá, letting it answer in English should recover the accuracy. We re-ran the full Nigeria set with questions still asked in each Nigerian language but answers requested in English. The effect is negligible: averaged over the eight models, the change from the native condition is +0.5 points for Pidgin, −0.9 for Hausa, −0.2 for Igbo, and +1.9 for

Yorùbá. Expression is not the bottleneck. A model that fails on a Nigerian fact asked in Yorùbá does not know the fact, and is not merely unable to say it in Yorùbá. If translation were the problem, removing it would have recovered the accuracy. Figure 6 compares the two conditions per model.

Does answering in English recover Nigerian facts?

Figure 6. Answering in English does not recover Nigerian facts. For each model, accuracy on Nigeria questions averaged over the four Nigerian languages when answering in-language (grey) versus in English (colour). The gaps are within ±2 points for every model: failures reflect missing knowledge, not an inability to express the answer in-language.

The language effect tracks web representation

Figure 3 plots per-language accuracy against each language’s Wikipedia article count (a proxy for web text availability; Figure 1). Averaged over models, the ordering English > Pidgin > Hausa > Igbo > Yorùbá is monotone on all three country sets. Two details sharpen the picture. First, Pidgin behaves unlike the other three. It has the fewest Wikipedia articles, about 1.6k, yet costs the least, because an English-derived creole is largely legible through English itself. Article counts understate its effective representation. Second, the slope depends on how much knowledge there is to lose: the English-to-Yorùbá drop is 22–24 points on the UK/US sets but only 9 points on Nigeria, where English accuracy is already low. Language degrades the retrieval of knowledge the model has; it cannot subtract knowledge the model lacks. Figure 7 in Appendix H breaks this down per model. Every model holds substantial UK/US knowledge in English, but how much survives a Nigerian query language varies sharply with capability: Claude Opus 5 and GPT-5.6 keep nearly all of it, Qwen3 and Mistral Large 3 keep about a third.

Model-averaged judged accuracy per query language against the language’s web representation

Figure 3. Model-averaged judged accuracy per query language against the language’s web representation (Wikipedia articles, log scale), for each country set. Accuracy rises with representation; Pidgin sits far above the trend for its article count, consistent with its legibility through English.

Do US / UK facts survive being asked in a Nigerian language?

Figure 7. What a Nigerian query language costs on facts the models demonstrably hold. For each model, accuracy on the UK (left) and US (right) sets when the question is asked in English (grey) versus averaged over the four Nigerian languages (colour). Unlike Figure 6, where the gaps vanish because the knowledge is missing in the first place, here the gaps are large and negative for the weaker models, and they widen as overall capability falls: Claude Opus 5 and GPT-5.6 carry foreign facts across languages nearly intact (−6 to +3 points), whereas Qwen3 and Mistral Large 3 lose roughly two thirds of theirs (−28 to −32). This is the language effect measured where there is knowledge to lose.

The deficit extends across the continent

We evaluated Claude Opus 5 on the 54 per-country sets of § 3.4. Figure 4 maps the result, and Table 9 in Appendix H lists every per-country accuracy.

CountrynnAcc.CountrynnAcc.CountrynnAcc.
Algeria12544.8Eswatini12433.9Niger12342.3
Angola12434.7Ethiopia12538.4Nigeria12536.0
Benin12544.8Gabon12543.2Rwanda12541.6
Botswana12533.6Ghana12545.6Senegal12536.0
Burkina Faso12339.8Guinea12552.8Seychelles11929.4
Burundi12545.6Guinea-Bissau12233.6Sierra Leone12545.6
Cabo Verde12332.5Kenya12551.2Somalia12541.6
Cameroon12542.4Lesotho12539.2South Africa12446.8
Central African Rep.11931.1Liberia12540.0South Sudan12534.4
Chad11828.8Libya12030.8Sudan12234.4
Comoros12337.4Madagascar12533.6São Tomé & Príncipe12536.8
Congo (Rep.)12537.6Malawi12539.2Tanzania12548.0
Côte d’Ivoire12541.6Mali12536.8The Gambia12245.9
Djibouti12437.1Mauritania12531.2Togo12141.3
DR Congo12531.2Mauritius12542.4Tunisia12441.1
Egypt12547.2Morocco12558.4Uganda12454.0
Equatorial Guinea12539.2Mozambique12439.5Zambia12538.4
Eritrea11937.8Namibia12242.6Zimbabwe12548.8

Table 9. Per-country factual accuracy of Claude Opus 5 on the NaijaFacts cross-country set, judged by an LLM grader and listed alphabetically. nn is the number of verified items for that country, which ranges from 118 to 125 because ambiguous items were dropped during verification, so accuracy is reported as a proportion. Overall: 2681/6689 = 40.1%.

The domain split is large. The model scores 95.8% on culture (689 items) and 97.7% on politics (642), but 26.0% on statistics (5,358). Frontier models hold the salient, narrative facts about African countries, meaning capitals, leaders, and landmarks, while the quantitative record of fertility, mortality, and economic indicators from national statistical offices and DHS surveys is almost entirely absent. A benchmark built from salient facts would conclude that models know Africa well, and grounded statistics show they do not.

Aggregate accuracy is 40.1% (2,681/6,689), but statistics is the bulk of the set, so that number is close to the statistics figure pulled up by two small, easy domains. Per country it ranges from 28.8% (Chad) to 58.4% (Morocco), and Nigeria, Africa’s most populous country, sits below the middle at 36.0%. Nigeria scores higher there than the 28% of Section 5 only because this set mixes in the two easy domains. The deficit is therefore continental and not Nigerian.

The same split appears inside the Nigeria set. Eleven items ask about the statistical system rather than a statistic, such as what TFR abbreviates, and they score 59.8% against 8.6% on the items drawn

Judged accuracy of Claude Opus 5 on country-grounded English questions across all 54 African countries

caption: Judged accuracy of Claude Opus 5 on country-grounded English questions across all 54 African countries (118–125 items each), spanning 28.8% (Chad) to 58.4% (Morocco), mean 40.1%.

from the 2024 NDHS. Those eleven are also the only Nigerian items with non-numeric answers, so salience and answer type cannot be separated here.

A stale training cut-off does not explain the deficit. Items about the 2006 census score 9.2%, no better than the 2024 items, so eighteen-year-old Nigerian statistics are absorbed no better than last year’s. Age is not what makes these facts hard.

A similar asymmetry has been measured outside Africa. LocalBench [9] reports 56.8% on narrative questions about US counties against under 15.5% on numerical ones, so part of what we see is a general difficulty with quantitative facts and not specific to Africa. It cannot be the whole story, though, because the UK and US sets beat every African country on comparable facts. Both effects are present, and the geographic one survives after the numeric one is accounted for.

We think the explanation is the form in which these statistics exist, not whether they exist at all. Every figure we grade is public, but it lives inside a PDF report table, and pretraining corpora are dominated by running prose, so a number can be openly available and still barely present in what a model absorbs. That would explain why contamination looks unlikely even though every source is open, and it predicts which fix works: retrieval over authoritative tables rather than more pretraining.

Conclusion

NaijaFacts is a multilingual benchmark that grounds factual questions about Nigeria in authoritative official statistics and separates two failure modes that translation-based benchmarks conflate: degradation in a low-resource language, and absence of the underlying fact. Asking the same questions about the UK and the US separates country familiarity from language resource level.

The knowledge effect dominates. Asked in English, models average 49–50% on US and UK official statistics but only 20% on the matched Nigerian statistics, and answering in English recovers essentially nothing, so most failures on Nigerian facts reflect missing knowledge and not translation.

Scale does not rescue it either: the strongest models reach 66–79% on the UK and US sets while no model exceeds 31% on Nigerian statistics. The language effect is real but smaller and tracks web representation, falling from English through Pidgin, Hausa, and Igbo to Yorùbá, and it costs most where models have knowledge to lose. Across all 54 African countries the profile is the same, with 96% accuracy on culture and 98% on politics against 26% on official statistics.

Serving African users therefore needs authoritative African data, the statistical record in particular, curated into training corpora, not only models adapted to African languages. We release the data, harness, and per-item outputs so the benchmark can be reproduced and extended to further countries.

Our study has a few limitations that future work should address. The Nigeria subset is 100 items per language drawn largely from a few source tables, so it probes grounded statistical knowledge and not broad knowledge, and the 54-country extension is English-only on a single model. Official statistics are also public, so contamination cannot be excluded, though the low accuracy on statistics suggests these tables are poorly absorbed in practice. The next steps follow: extending the multilingual design beyond Nigeria, and testing whether injecting statistical tables during training moves the number that language adaptation leaves fixed.

Prompt templates

Every model sees the same two prompts in every condition. Placeholders in braces are substituted per item: {language} names the question’s language (English, Hausa, Yoruba, Igbo, or Nigerian Pidgin) and {question} carries the question text in that language.

System prompt (all conditions)

You are a factual question-answering assistant. IMPORTANT: Reply with ONLY the answer itself, no explanations, no preamble, no ‘The answer is...’, no additional context. For numbers: give just the number (e.g., ‘200 million’ or ‘45.2%’). For names/entities: give just the name (e.g., ‘Abuja’ or ‘National Population Commission’).

User prompt (native condition)

Answer this question in {language}: {question} Output format: Give only the answer, nothing else.

The answer-in-English condition (§ 6.1) changes one word of the user prompt, so that the first line reads “Answer this question in English:” while the question itself stays in the target language. Everything else, including the system prompt, is unchanged.

LLM-as-judge prompt

The judge (Claude Opus 5) receives the system prompt below and returns a single-token verdict. The user turn supplies the question, the gold answer, and the model’s answer.

Judge system prompt

You are an evaluator assessing whether a model’s answer is factually correct compared to the ground truth. Your task: Determine if the model’s answer contains the same core factual information as the expected answer.

IMPORTANT – Mark as CORRECT if:

– The core fact matches, even with different wording – Numbers are the same or very close (89% = 88.9%, 59597542 = 59,597,300) – Minor additions/omissions of words like “The”, “a”, “an”, articles – The meaning is the same even if the actual words are different – Missing or extra punctuation (periods, commas) – Missing units when the number is correct (“1.41” = “1.41 children per woman”) – Abbreviations match full names (“ONS” = “Office for National Statistics”) – Extra context is provided but core answer is correct

Mark as INCORRECT ONLY if:

– The factual content is genuinely wrong – The answer is empty or refuses to answer – A completely different fact is stated

Examples of CORRECT matches:

– “Office for National Statistics” vs “The Office for National Statistics.” – “Every 10 years” vs “Every 10 years.” – “89%” vs “88.9%” – “56,489,800” vs “56,489,800 people.” – “Census Act 1920” vs “The Census Act 1920.” – “London” vs “London.”

Respond with ONLY: CORRECT or INCORRECT

For the native condition the judge compares in the answer’s language, and for the answer-in-English condition it compares in English. On the 54-country sets the user turn carries one extra block, the item’s grading notes, which name the answer variants to accept and the plausible-sounding answers the sources positively contradict. Those notes override the defaults above, so an answer the notes rule out is marked incorrect even when it looks reasonable (Section 4).

Example items

Table 2 shows one NaijaFacts item in all five languages. Every item carries the same gold answer across languages, so any accuracy difference between languages on the same item is attributable to the query language alone. Diacritics are reproduced from the source data. Table 3 shows a numeric item in the same format, together with its gold answer in each language and how many models answered it correctly, which reproduces the paper’s main language ordering within a single item.

LanguageQuestion (gold answer: National Population Commission)
EnglishWhat does NPC stand for in Nigeria’s population statistics system?
PidginWetin be di NPC stand for for Naija population statistics?
YorùbáKini NPC duro fun ninu eto iṣiro awon eniyan Naijiria?
IgboGịnị ka NPC pụtara n’usoro ọnụ ọgụgụ ndị mmadụ nke Naịjịria?
HausaMenene ma’anar NPC a tsarin kididdigar yawan jama’a na Najeriya?

Table 2. An example NaijaFacts item across the five languages. Most items have numeric gold answers drawn from official statistics (e.g. “How many population classifications were used in the 2006 Nigeria census?” with gold answer “Eight”); entity items such as this one are included for coverage of the statistical system itself.

LanguageQuestionGold answerModels correct
EnglishWhat age range was used for all Nigerian women interviewed in the 2024 NDHS?15 to 49 years.8/8
PidginWetin be di age range wey dem use for all Naija wumen wey dem interview for 2024 NDHS?15 to 49 years.7/8
HausaWane adadin shekarun matan Najeriya aka yi wa tambayoyi a NDHS 2024?Shekaru 15 zuwa 49.6/8
IgboKedu oke afọ e jiri maka ụmụ nwanyi Naịjiria niile ndi a gbara ajụjụ ọnụ na NDHS 2024?Afọ 15 ruo 49.5/8
YorùbáKini ìwọ̀n ọjọ́-ori àwọn obinrin ni Naijiria ti wọn ba sọ̀rọ̀ ninu NDHS 2024?Ọ̀dún 15 si 49.4/8

Table 3. A numeric item in all five languages, with the gold answer as it appears in each language and the number of the eight evaluated models judged correct on it. The fact is identical in every row, and the answer is a short figure that leaves little room for expression difficulty, yet the count falls monotonically in exactly the order reported in § 5.1: English, Pidgin, Hausa, Igbo, Yorùbá. Note that the gold answer is itself translated, so numerals stay Arabic while the surrounding words change (Ọ̀dún, Afọ, Shekaru); the judge accepts any of these forms.

Dataset summary

Table 4 summarises NaijaFacts and its cross-country extension. The two halves are matched in different ways, which is worth keeping in mind when reading any comparison between them. The Nigeria, UK and US sets ask the same 100 questions of all three countries, so a difference between countries is attributable to the answer alone. The 54-country extension cannot work that way, because a question about a national census has no counterpart in a country that has never run one, so it instead holds the domain mix and answer-type distribution constant and draws each country’s answers from its own sources.

Models evaluated

Table 5 lists the eight models, their origins and their identifiers. The frontier set spans three geographical origins so that we can ask whether a model’s origin affects whose facts it holds (§ 5.1), and the two Nigeria-adapted models are both Llama-3-8B fine-tunes, which lets us separate the effect of language adaptation from the effect of scale. Parameter counts for the proprietary models are undisclosed. Table 6 then gives the model-averaged country by language matrix discussed in § 5.2.

Benchmark comparison

Table 7 compares NaijaFacts against the nearest related benchmarks along the three properties discussed in § 2: grounded statistical answers, Nigerian-language coverage, and a matched country ×\times language design. No prior benchmark combines all three, which is the gap NaijaFacts fills. The nearest neighbours each supply one or two: WorldBench varies the country but uses indicator databases and English only, Afri-MCQA authors natively in African languages but grounds its answers in everyday culture, and LocalBench grounds answers in official statistics but for US counties in English.

Sources and provenance

Table 8 gives the per-country breakdown. All sources are public official statistics. The 54-country extension instead draws on the World Bank World Development Indicators and the DHS Program APIs (§ 3.4); one example of the instrument disagreement that forces each indicator to a single source is that WDI and national DHS reports differ on total fertility rates.

Per-language and per-model results

Three tables carry the full numbers behind § 5 and § 6: Table 1 the per-model by per-language breakdown on the Nigeria set, Table 6 the model-averaged country by language matrix, and Table 9 the per-country accuracies for all 54 African countries.

The figures below add the per-model detail the body figures average away. Figure 5 gives the language effect model by model, Figure 6 the answer-in-English comparison of § 6.1, and Figure 7 the same language effect measured on the UK and US sets, where the models demonstrably hold the knowledge. Per-item model outputs and judge verdicts for every run are released with the code and data.

References

  1. [1]David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, et al. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
  2. [2]Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, and Paul Pu Liang. Language models’ factuality depends on the language of inquiry. arXiv preprint arXiv:2502.17955, 2025.arxiv.org/abs/2502.17955
  3. [3]Awarri Technologies, Federal Ministry of Communications, Innovation and Digital Economy, and National Centre for Artificial Intelligence and Robotics. N-ATLaS-LLM: A multilingual African language model. https://huggingface.co/NCAIR1/N-ATLaS, 2025. Model card. Llama-3 8B fine-tune with native support for English, Hausa, Igbo and Yoruba; the card reports roughly 200,000 instruction samples for each of the three Nigerian languages. Accessed: 10 September 2026.
  4. [4]Lucas Bandarkar, Davis Liang, Benjamin Muller, et al. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
  5. [5]Edward Bayes, Israel Abebe Azime, Jesujoba O. Alabi, et al. Uhura: A benchmark for evaluating scientific question answering and truthfulness in low-resource African languages. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 1485–1504, 2026.arxiv.org/abs/2412.00948
  6. [6]Nitay Calderon, Eyal Ben-David, Zorik Gekhman, Eran Ofek, and Gal Yona. Empty shelves or lost keys? recall is the bottleneck for parametric factuality. arXiv preprint arXiv:2602.14080, 2026.arxiv.org/abs/2602.14080
  7. [7]David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. Ethnologue: Languages of the World. SIL International, Dallas, Texas, 27 edition, 2024. First-language and second-language speaker estimates for Hausa, Yoruba, Igbo and Nigerian Pidgin. Online version: https://www.ethnologue.com. Accessed: 10 September 2026.
  8. [8]Federal Ministry of Health and Social Welfare of Nigeria (FMoHSW), National Population Commission (NPC) [Nigeria], and ICF. Nigeria Demographic and Health Survey 2024. https://dhsprogram.com/publications/publication-FR395-DHS-Final-Reports.cfm, 2025. Final report FR395. Source for 81 of the 100 Nigerian items.
  9. [9]Zihan Gao, Yifei Xu, and Jacob Thebault-Spieker. LocalBench: Benchmarking LLMs on county-level local knowledge and reasoning. arXiv preprint arXiv:2511.10459, 2025.arxiv.org/abs/2511.10459
  10. [10]Jacaranda Health. AfroLlama3. https://huggingface.co/Jacaranda/AfroLlama_V1, 2024. Model card. Llama-3 8B fine-tune covering Swahili, Xhosa, Zulu, Yoruba, Hausa and English, with no Igbo adaptation. Accessed: 10 September 2026.
  11. [11]Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online, November 2020. Association for Computational Linguistics.arxiv.org/abs/2010.06189
  12. [12]Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online, July 2020. Association for Computational Linguistics.arxiv.org/abs/2004.09095
  13. [13]Nora Kassner, Philipp Dufter, and Hinrich Schütze. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online, April 2021. Association for Computational Linguistics.arxiv.org/abs/2102.00894
  14. [14]Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, et al. MADLaD-400: A multilingual and document-level large audited dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.arxiv.org/abs/2309.04662
  15. [15]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanah Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics.
  16. [16]Mazda Moayeri, Elham Tabassi, and Soheil Feizi. WorldBench: Quantifying geographic disparities in LLM factual recall. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2024.DOI
  17. [17]Junho Myung, Nayeon Lee, Yi Zhou, et al. BLEnD: A benchmark for llms on everyday knowledge in diverse cultures and languages. In Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.09948
  18. [18]National Bureau of Statistics (Nigeria). Official statistical releases. https://www.nigerianstat.gov.ng, 2026. Source for the eleven Nigerian items that name no census or survey edition. Accessed: 10 September 2026.
  19. [19]National Center for Health Statistics (US). National center for health statistics data releases, reference years 2022–2024. https://www.cdc.gov/nchs, 2026. Source for the US vital statistics and health items. Accessed: 10 September 2026.
  20. [20]National Population Commission (Nigeria). Population and housing census documentation, 2006 and 2023 editions. https://nationalpopulation.gov.ng, 2026. Three Nigerian items draw on the 2006 Population and Housing Census; five draw on the documented design of the 2023 census exercise, its enumeration-area procedure and questionnaire sections rather than any enumeration outcome. Accessed: 10 September 2026.
  21. [21]Office for National Statistics (UK) and National Records of Scotland. Official statistical releases, reference years 2021–2025. https://www.ons.gov.uk, 2026. UK gold answers. Census 2021 for England and Wales and Scotland’s Census 2022 supply the census items; the remainder are ONS mid-year estimates and topic releases with reference years 2021 to 2025. The release URL for each item ships with the benchmark. Accessed: 10 September 2026.
  22. [22]Odunayo Ogundepo, Tajuddeen R. Gwadabe, Clara E. Rivera, et al. AfriQA: Cross-lingual open-retrieval question answering for African languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14957–14972, Singapore, December 2023. Association for Computational Linguistics.arxiv.org/abs/2305.06897
  23. [23]Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, et al. AfroBench: How good are large language models on African languages? In Findings of the Association for Computational Linguistics: ACL 2025, 2025.
  24. [24]OpenAI. GPT-4 technical report. ArXiv, abs/2303.08774, 2023.arxiv.org/abs/2303.08774
  25. [25]Jirui Qi, Raquel Fernández, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666, Singapore, December 2023. Association for Computational Linguistics.DOI
  26. [26]Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 311–325, Mexico City, Mexico, June 2024. Association for Computational Linguistics.arxiv.org/abs/2308.10168
  27. [27]Atnafu Lambebo Tonja, Srija Anand, Emilio Villa-Cueva, et al. Afri-MCQA: Multimodal cultural question answering for African languages. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 40249–40282, 2026.
  28. [28]U.S. Census Bureau. Decennial census and american community survey releases, reference years 2020–2025. https://www.census.gov, 2026. US gold answers. The 2020 Decennial Census supplies the census items; the remainder are American Community Survey and population-estimate releases. The release URL for each item ships with the benchmark. Accessed: 10 September 2026.
  29. [29]Jason Wei, Karina Nguyen, Hyung Won Chung, et al. Measuring short-form factuality in large language models (SimpleQA), 2024.arxiv.org/abs/2411.04368
  30. [30]Wikimedia Foundation. List of wikipedias. https://meta.wikimedia.org/wiki/List_of_Wikipedias, 2026. Article counts as of the snapshot accessed 24 August 2026.
  31. [31]World Health Organization and UNICEF. Who/unicef joint monitoring programme for water supply, sanitation and hygiene. https://washdata.org, 2025. Source for the UK and US water and sanitation items. Accessed: 10 September 2026.
  32. [32]Da Yin, Hritik Bansal, Masoud Monajatipooor, Liunian Harold Li, and Kai-Wei Chang. Ge-oMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Paper details

vlnc:W_5ccttgg7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents