Back to Gap Map Model components / Language-specific evaluation sets

DarijaMMLU

Mohamed bin Zayed University of Artificial Intelligence
open / Overall score: 2.0

DarijaMMLU tests language models on 22,027 multiple-choice questions in Moroccan Darija across 44 subjects. The questions are selected subsets of MMLU and ArabicMMLU, machine-translated into Darija with Claude 3.5 Sonnet and then manually reviewed and adapted, with technical and culturally inappropriate topics left out. MBZUAI-Paris built it as part of the Atlas-Chat evaluation suite.

Openness

5 medium confidence
5.0
license
mit (card metadata; Licensing Information links MMLU's LICENSE, although part of the questions come from ArabicMMLU, released for noncommercial use)
access
public (ungated on the Hub)
dataset_card
present (card describes sources, translation model and review)

The questions download freely and the card states where each one came from and how it was translated. Whether the permissive grant can cover the ArabicMMLU-derived questions is unclear, since the source benchmark forbids commercial use.

Adoption

2 high confidence
2.0

Hugging Face downloads of the Hub repository. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.

Capability

not assessed

DarijaMMLU's questions were machine-translated and then reviewed by people, but no source says how many were reviewed, so its depth cannot be counted and the map shows no capability score for now. Its overall score reflects adoption alone.

Verified 2026-09-24