AfroBench
McGill NLPAfroBench is an evaluation suite for large language models covering 64 African languages, 15 tasks and 22 datasets. It carries AfriMMLU, AfriXNLI, AfriMGSM, AfriSenti, MasakhaNER, MasakhaNEWS, MasakhaPOS, AfriQA, MAFAND-MT, SALT, Uhura, InjongoIntent, NaijaRC, NollySenti, NTREX African and AfriADR, with lm-evaluation-harness configs and a public leaderboard. It is published from the McGill-NLP GitHub organization, with its data in a Masakhane collection.
Belebele, FLORES, SIB-200, MMMLU and XL-Sum are in the suite but are separate entries, so their repositories are not listed or counted here. NaijaRC and NollySenti sit in a personal Hugging Face namespace and are also left out.
Openness
4 medium confidence- license
- mixed-per-subset (afrimmlu: apache-2.0; afrixnli: apache-2.0; afrimgsm: apache-2.0; afrisenti: cc-by-nc-sa-2.0; masakhaner-x: unknown; masakhanews: afl-3.0; masakhapos: afl-3.0; afriqa-gold-passages: cc-by-sa-4.0; mafand: cc-by-nc-4.0; uhura-arc-easy: mit; InjongoIntent: apache-2.0; ntrex_african: cc-by-sa-4.0; AfriADR: apache-2.0; salt: cc-by-sa-4.0; the GitHub repository has no LICENSE file)
- access
- public (Masakhane member repositories ungated; Sunbird/salt behind an automatic gate)
- dataset_card
- present (README, project site and paper describe tasks and datasets; each member has its own card)
Each member dataset keeps its own license, from Apache-2.0 and MIT to non-commercial terms on MAFAND-MT and AfriSenti, and MasakhaNER-X lists its license as unknown. Everything but SALT downloads without a gate.
- https://huggingface.co/api/collections/masakhane/afrobench-67dbf553ebf5701c2207f883 recorded 2026-09-24
Collection items include masakhane/afrimmlu, afrixnli, afrimgsm, afrisenti, masakhaner-x, masakhanews, masakhapos, afriqa-gold-passages, mafand, uhura-arc-easy, InjongoIntent, ntrex_african, AfriADR and Sunbird/salt.
- https://huggingface.co/api/datasets/masakhane/afrisenti recorded 2026-09-24
cardData license "cc-by-nc-sa-2.0".
- https://huggingface.co/api/datasets/masakhane/mafand recorded 2026-09-24
cardData license ["cc-by-nc-4.0"]; "gated": false.
- https://huggingface.co/api/datasets/masakhane/masakhaner-x recorded 2026-09-24
cardData license ["unknown"].
- https://huggingface.co/api/datasets/Sunbird/salt recorded 2026-09-24
"gated": "auto"; cardData license "cc-by-sa-4.0".
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/README.md recorded 2026-09-24
"All scripts, prompts and datasets are opensource and available"; run via LM-Harness or API; no license section.
Adoption
1 low confidenceGitHub stars on the AfroBench repository. The member datasets it evaluates are also downloaded on their own, but those downloads are not runs of the suite, so they are left out; a star is attention rather than use.
- https://ungh.cc/repos/McGill-NLP/AfroBench recorded 2026-09-24
GitHub repository record for McGill-NLP/AfroBench (via the ungh.cc mirror of the GitHub API): 24 stargazers, last push 2025-07-28.
Capability
not assessedIn the typical language AfroBench targets, its deepest test has four hundred to sixteen hundred items, but in 29 of its 64 languages that test comes from another dataset with nothing independent in the language to back it, and 5 languages have no countable test at all. Whether the typical language has fewer than a hundred corroborated items or four hundred or more depends on where those 5 would land, so the map shows no capability score for now and its overall score reflects adoption alone.
- https://arxiv.org/abs/2207.04672 recorded 2026-11-18
"we evaluated the performance of over 40,000 different translation directions using a human-translated benchmark, Flores-200"
- https://arxiv.org/abs/2309.07445 recorded 2026-11-18
"The dataset is based on Flores-200 machine translation corpus. We annotated the English portion of the dataset and extended the sentence-level annotation to the remaining 203 languages covered in the corpus."
- https://arxiv.org/abs/2406.03368 recorded 2026-11-18
"we introduce IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based question answering~(AfriMMLU)"
- https://arxiv.org/abs/2408.17024 recorded 2026-11-18
InkubaLM abstract reports performance on "AfriMMLU, and the AfriXnli task".
- https://arxiv.org/abs/2502.09814 recorded 2026-11-18
"a multicultural, open-source benchmark dataset for 16 African languages with utterances generated by native speakers"
- https://arxiv.org/html/2305.06897 recorded 2026-11-18
"For each of the 10 languages in AfriQA, a team of 2–6 native speakers was responsible for the data collection and annotation."; "The annotation pipeline consisted of 4 distinct stages: 1) question elicitation in an African language; 2) translation of questions into a pivot language"; "For Swahili, questions elicited in TyDi QA, which remained unanswered in the original dataset were used with light curation from the Swahili team for correctness."
- https://arxiv.org/html/2308.16884 recorded 2026-11-18
"The passages in the Belebele corpus are directly sourced from the FLoRes-200 Machine Translation Benchmark"; "Translators were instructed to maintain informative and standardized content"
- https://arxiv.org/html/2311.07978 recorded 2026-11-18
Table 2 per-language size ("Per. Lang." column): SIB-200 204, NollySenti 500, Injongo-Intent 640, AfriXNLI 600, Belebele 900, AfriMMLU 500, MMMLU 14042, AfriMGSM 250, Flores-200 1012, NTREX 2000, Salt 500, ranges for the rest and "excl. amh: 500 & luo: 185 (in MasakhaNER-X)"; Table 3 AfriADR test sizes Ghomálá’ 1430, Fon 1579, Igbo 1500, Wolof 1500, Yorùbá 1558; Table 11 marks the languages each dataset covers; "We focus on two human-translated MMLU datasets"; Uhura is "the human translation of scientific Arc-Easy benchmark"; "To create AfriADR , we selected the five languages with extensive use of diacritics from Mafand MT dataset, then, we strip all accents and diacritics on each sentence"
- https://arxiv.org/html/2406.03368 recorded 2026-11-18
IrokoBench's Table 1 has AfriMGSM in "17 (excl. Swahili, inc. Vai)" languages with 250 test questions; "We recruited language coordinators for each of the 16 African languages and French, and asked them to recruit professional translators to translate the sentences."; "Most of the translators translated from English except for Ewe, Lingala and Wolof translators that translated from French"; "We extend our gratitude to Charles Riley for facilitating our connection with the Vai translator for the MGSM data translation."; "Here, we human translate the English portion of XNLI (a multilingual dataset comprising 15 languages, including Swahili) into the 15 African languages (excluding Swahili)"; "AfriMGSM expands the original MGSM dataset (Shi et al., 2023), which contains 250 QA pairs and 11 languages (including Swahili), to 15 more languages."
- https://arxiv.org/pdf/2205.02022 recorded 2026-11-18
MAFAND-MT's Table 1 gives the news source and data splits for "six languages (bam, bbj, ewe, fon, mos, wol) spoken predominantly in Francophone Africa and five languages (lug, luo, pcm, tsn, twi)", with test splits of 1600, 1430, 1563, 1579, 1574 and 1500 for the six and 1500, 1500, 1564, 1500 and 1500 for the five, the news crawled from local newspapers publishing in French or English; step 2 of its creation is "Translation of 5k–8k sentences by professional translators". Its Hausa, Igbo, Swahili, Yorùbá and isiZulu splits come from existing corpora (Appendix B), which it does not say people translated
- https://arxiv.org/pdf/2210.03057 recorded 2026-11-18
MGSM: "We enlisted the help of paid professional translators (two for Chinese and German, three for Russian, five for Thai, one for each remaining target language) for the manual translation of the 250 selected English-language examples from GSM8K"; Swahili is among its languages ("underrepresented languages such as Bengali and Swahili")
- https://arxiv.org/pdf/2305.10971 recorded 2026-11-18
"We hire professional translators in Nigeria and ask them to translate 1,010 reviews randomly chosen from the 1,900 English reviews."; "Thus, we have a parallel review dataset in English and other Nigerian languages"; Table 2 gives 500 test reviews each for Hausa, Igbo, "Naija (pcm)" and Yorùbá
- https://arxiv.org/pdf/2305.13989 recorded 2026-11-18
MasakhaPOS: "we make use of MAFAND-MT (Adelani et al., 2022a) translation corpus that is based on the news domain"; its Table 1 gives "MAFAND-MT" as the source for Bambara, Ghomálá’, Éwé, Fon, Luganda, Luo, Mossi, Naija, Setswana, Akan/Twi and Wolof and a newspaper or broadcaster for the other nine languages, for which "We collected monolingual news corpus with an open license for about eight African languages, mostly from local newspapers"
- https://datasets-server.huggingface.co/size?dataset=Davlan/nollysenti recorded 2026-11-18
the Hub's split sizes for Davlan/nollysenti: 500 test rows in each of the en, ha, ig, pcm and yo configs
- https://datasets-server.huggingface.co/size?dataset=Davlan/sib200 recorded 2026-11-18
the Hub's split sizes for Davlan/sib200: 204 test rows in each of the afr_Latn, sot_Latn and taq_Latn configs
- https://datasets-server.huggingface.co/size?dataset=facebook/belebele recorded 2026-11-18
the Hub's split sizes for facebook/belebele: 900 test rows in each of the fuv_Latn, gaz_Latn, kea_Latn, kin_Latn, lug_Latn, luo_Latn, nya_Latn, plt_Latn, som_Latn, sot_Latn, ssw_Latn and tsn_Latn configs
- https://datasets-server.huggingface.co/size?dataset=masakhane/afrimgsm recorded 2026-11-18
the Hub's split sizes for masakhane/afrimgsm: 250 test rows in every language config, vai and wol among them
- https://datasets-server.huggingface.co/size?dataset=masakhane/afrisenti recorded 2026-11-18
the Hub's split sizes for masakhane/afrisenti: test rows per language config, 37,670 in all, the total AfroBench's Table 2 gives
- https://datasets-server.huggingface.co/size?dataset=masakhane/InjongoIntent recorded 2026-11-18
the Hub's split sizes for masakhane/InjongoIntent: 640 test rows per language config, 638 for wol and 622 for eng
- https://datasets-server.huggingface.co/size?dataset=masakhane/masakhanews recorded 2026-11-18
the Hub's split sizes for masakhane/masakhanews: test rows per language config, 6,242 in all, the total AfroBench's Table 2 gives
- https://datasets-server.huggingface.co/size?dataset=masakhane/masakhapos recorded 2026-11-18
the Hub's split sizes for masakhane/masakhapos: test rows per language config, 12,190 in all, the total AfroBench's Table 2 gives
- https://datasets-server.huggingface.co/size?dataset=masakhane/ntrex_african recorded 2026-11-18
the Hub's split sizes for masakhane/ntrex_african: 1997 test rows in each language config
- https://datasets-server.huggingface.co/size?dataset=masakhane/uhura-arc-easy recorded 2026-11-18
the Hub's split sizes for masakhane/uhura-arc-easy: test rows per *_multiple_choice config, 3,257 in all, the total AfroBench's Table 2 gives
- https://huggingface.co/datasets/masakhane/afrisenti recorded 2026-11-18
"covering 110,000+ annotated tweets in 14 African languages"
- https://huggingface.co/datasets/masakhane/mafand recorded 2026-11-18
"MAFAND was created from the news domain, translated from English or French to an African language"
- https://huggingface.co/datasets/masakhane/masakhanews recorded 2026-11-18
"MasakhaNEWS is the largest publicly available dataset for news topic classification in 16 languages widely spoken in Africa."
- https://huggingface.co/datasets/masakhane/masakhapos recorded 2026-11-18
"The source of the data is from the news domain"
- https://mcgill-nlp.github.io/AfroBench/ recorded 2026-11-18
"Evaluating 64 African languages across 15 NLP tasks and 22 Datasets"; leaderboard and lm-eval-harness scripts.
- https://raw.githubusercontent.com/facebookresearch/belebele/main/README.md recorded 2026-11-18
"900 questions per language variant"
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/bem/gold_span_passages.afriqa.bem.en.test.json recorded 2026-11-18
the Bemba gold-passage test split the AfriQA loader reads: 309 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/fon/gold_span_passages.afriqa.fon.fr.test.json recorded 2026-11-18
the Fon gold-passage test split the AfriQA loader reads: 386 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/hau/gold_span_passages.afriqa.hau.en.test.json recorded 2026-11-18
the Hausa gold-passage test split the AfriQA loader reads: 300 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/ibo/gold_span_passages.afriqa.ibo.en.test.json recorded 2026-11-18
the Igbo gold-passage test split the AfriQA loader reads: 409 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/kin/gold_span_passages.afriqa.kin.en.test.json recorded 2026-11-18
the Kinyarwanda gold-passage test split the AfriQA loader reads: 347 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/swa/gold_span_passages.afriqa.swa.en.test.json recorded 2026-11-18
the Swahili gold-passage test split the AfriQA loader reads: 295 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/twi/gold_span_passages.afriqa.twi.en.test.json recorded 2026-11-18
the Twi gold-passage test split the AfriQA loader reads: 486 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/yor/gold_span_passages.afriqa.yor.en.test.json recorded 2026-11-18
the Yoruba gold-passage test split the AfriQA loader reads: 254 records, one per line
- https://raw.githubusercontent.com/masakhane-io/afriqa/main/data/gold_passages/zul/gold_span_passages.afriqa.zul.en.test.json recorded 2026-11-18
the Zulu gold-passage test split the AfriQA loader reads: 325 records, one per line
- https://raw.githubusercontent.com/masakhane-io/lafand-mt/main/data/json_files/fr-bam/test.json recorded 2026-11-18
the Bambara test split MAFAND-MT publishes and AfroBench's mafand task loads: 1,500 French-Bambara sentence pairs, one per line, where the paper's Table 1 gives 1,600
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/afrimgsm.yaml recorded 2026-11-18
AfroBench's afrimgsm task: "dataset: masakhane/afrimgsm", "test_split: test", languages including "vai" and "wol"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/afriqa.yaml recorded 2026-11-18
AfroBench's afriqa task: "dataset: masakhane/afriqa-gold-passages", "test_split: test", languages "bem", "fon", "hau", "ibo", "kin", "swa", "twi", "yor" and "zul"; "The question is in {language}, while the context is in English or French"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/belebele.yaml recorded 2026-11-18
AfroBench's belebele task: "dataset: facebook/belebele", "test_split: test", and languages including "fuv_Latn", "gaz_Latn", "kea_Latn", "kin_Latn", "lug_Latn", "luo_Latn", "nya_Latn", "plt_Latn", "som_Latn", "sot_Latn", "ssw_Latn" and "tsn_Latn"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/flores.yaml recorded 2026-11-18
AfroBench's flores task: "dataset: facebook/flores", "test_split: devtest", and pairs including "fuv_Latn-eng_Latn" and "tso_Latn-eng_Latn"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/mafand.yaml recorded 2026-11-18
AfroBench's mafand task: "dataset: masakhane/mafand", "test_split: test", and pairs including "fr-bam", "fr-bbj", "fr-ewe", "fr-fon", "en-lug", "en-luo", "fr-mos", "en-pcm", "en-tsn", "en-twi" and "fr-wol"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/nollysenti.yaml recorded 2026-11-18
AfroBench's nollysenti task: "dataset: Davlan/nollysenti", "test_split: test", languages "en", "ha", "ig", "pcm" and "yo"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/prompt_with_API/afrobench_tasks/sib.yaml recorded 2026-11-18
AfroBench's sib task: "dataset: Davlan/sib200", "test_split: test", and languages including "afr_Latn", "sot_Latn" and "taq_Latn"
- https://raw.githubusercontent.com/McGill-NLP/AfroBench/main/README.md recorded 2026-11-18
Run with "LM-Harness (lm_eval/tasks/afrobench)"; results submitted to "the official leaderboard".
- https://raw.githubusercontent.com/MicrosoftTranslator/NTREX/main/README.md recorded 2026-11-18
"human translated references for the original 128 languages"
Verified 2026-09-24