DarijaMMLU
Mohamed bin Zayed University of Artificial IntelligenceDarijaMMLU tests language models on 22,027 multiple-choice questions in Moroccan Darija across 44 subjects. The questions are selected subsets of MMLU and ArabicMMLU, machine-translated into Darija with Claude 3.5 Sonnet and then manually reviewed and adapted, with technical and culturally inappropriate topics left out. MBZUAI-Paris built it as part of the Atlas-Chat evaluation suite.
Openness
5 medium confidence- license
- mit (card metadata; Licensing Information links MMLU's LICENSE, although part of the questions come from ArabicMMLU, released for noncommercial use)
- access
- public (ungated on the Hub)
- dataset_card
- present (card describes sources, translation model and review)
The questions download freely and the card states where each one came from and how it was translated. Whether the permissive grant can cover the ArabicMMLU-derived questions is unclear, since the source benchmark forbids commercial use.
- https://huggingface.co/api/datasets/MBZUAI-Paris/DarijaMMLU recorded 2026-09-24
API JSON: "gated": false, "private": false.
- https://huggingface.co/datasets/MBZUAI-Paris/DarijaMMLU/raw/main/README.md recorded 2026-09-24
Card metadata "license: mit"; Licensing Information: "[MIT License](https://github.com/hendrycks/test/blob/master/LICENSE)"; each row has a "source" field (MMLU or ArabicMMLU).
Adoption
2 high confidenceHugging Face downloads of the Hub repository. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.
- https://huggingface.co/api/datasets/MBZUAI-Paris/DarijaMMLU recorded 2026-09-24
3008 downloads in the trailing 30 days for MBZUAI-Paris/DarijaMMLU
Capability
not assessedDarijaMMLU's questions were machine-translated and then reviewed by people, but no source says how many were reviewed, so its depth cannot be counted and the map shows no capability score for now. Its overall score reflects adoption alone.
- https://arxiv.org/abs/2409.17912 recorded 2026-11-08
Abstract: "our 9B model gains a 13% performance boost over a larger 13B model on DarijaMMLU, in our newly introduced evaluation suite for Darija".
- https://datasets-server.huggingface.co/size?dataset=MBZUAI-Paris/DarijaMMLU recorded 2026-11-08
the Hub's split sizes for the 44 subject configs, summed: test 21,792 and dev 235 (22,027 in all, the card's total); there is no combined config
- https://huggingface.co/datasets/MBZUAI-Paris/DarijaMMLU/raw/main/README.md recorded 2026-11-08
"The selected subsets were translated into Darija using Claude 3.5 Sonnet." "created through a combination of machine translation and manual review". Card metadata lists language_creators "machine-translated"; "The translations were produced using machine translation with manual curation for quality control."
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/darijammlu/_default_darijammlu_template_yaml recorded 2026-11-08
the harness's DarijaMMLU template: "dataset_path: MBZUAI-Paris/DarijaMMLU", "test_split: test", "fewshot_split: dev"
Verified 2026-09-24