ArabicMMLU
Mohamed bin Zayed University of Artificial IntelligenceArabicMMLU tests language models on 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, drawn from school exams in countries of North Africa, the Levant, and the Gulf. Ten native speakers from several countries built it, grouping questions into STEM, social science, humanities, Arabic language, and other subjects. MBZUAI leads the work with university and industry partners.
Openness
2 medium confidence- license
- cc-by-nc-4.0 (Hub card metadata; the GitHub README License section says CC BY-NC-SA 4.0)
- access
- public (ungated on the Hub; also in the GitHub data folder)
- dataset_card
- present (card and README describe sources, construction and statistics)
Anyone can download the questions and the evaluation code, and the card and paper explain how they were collected. The noncommercial license keeps it out of commercial evaluation pipelines, and the Hub and GitHub copies disagree on whether a share-alike condition applies.
- https://huggingface.co/api/datasets/MBZUAI/ArabicMMLU recorded 2026-09-24
API JSON: "gated": false, "private": false.
- https://huggingface.co/datasets/MBZUAI/ArabicMMLU/raw/main/README.md recorded 2026-09-24
Card metadata: "license: cc-by-nc-4.0".
- https://raw.githubusercontent.com/mbzuai-nlp/ArabicMMLU/main/README.md recorded 2026-09-24
"The ArabicMMLU dataset is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License."
Adoption
2 high confidenceHugging Face downloads of the Hub copy. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations. It misses copies taken from the GitHub data folder.
- https://huggingface.co/api/datasets/MBZUAI/ArabicMMLU recorded 2026-09-24
3727 downloads in the trailing 30 days for MBZUAI/ArabicMMLU
Capability
5 medium confidenceIn its one target language, ArabicMMLU's deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.
- https://arxiv.org/abs/2402.12840 recorded 2026-11-08
Abstract: "the first multi-task language understanding benchmark for the Arabic language, sourced from school exams ... 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA)".
- https://arxiv.org/html/2402.12840 recorded 2026-11-08
knowledge evaluation for Arabic models "primarily conducted using datasets translated from English to Arabic"; ArabicMMLU is "sourced from school exams across Arabic-speaking countries"; filtering reduced "the initial set of over 15,000 questions to 14,575 unique questions"
- https://datasets-server.huggingface.co/size?dataset=MBZUAI/ArabicMMLU recorded 2026-11-08
the Hub's split sizes for the "All" config: "split":"test" "num_rows":14455 and "split":"dev" "num_rows":120; the 40 subject configs sum to the same totals
- https://huggingface.co/datasets/MBZUAI-Paris/DarijaMMLU/raw/main/README.md recorded 2026-11-08
"translated from selected subsets of the Massive Multitask Language Understanding (MMLU) and ArabicMMLU benchmarks".
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/arabicmmlu/_default_arabicmmlu_template_yaml recorded 2026-11-08
the harness's ArabicMMLU template: "dataset_path: MBZUAI/ArabicMMLU", "test_split: test", "fewshot_split: dev"
- https://raw.githubusercontent.com/mbzuai-nlp/ArabicMMLU/main/README.md recorded 2026-11-08
"The data construction process involved a total of 10 Arabic native speakers from different countries"; "These questions are written in Arabic."
Verified 2026-09-24