Back to Gap Map Model components / Language-specific evaluation sets

ArabicMMLU

Mohamed bin Zayed University of Artificial Intelligence
restricted / Overall score: 4.1(strong)

ArabicMMLU tests language models on 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, drawn from school exams in countries of North Africa, the Levant, and the Gulf. Ten native speakers from several countries built it, grouping questions into STEM, social science, humanities, Arabic language, and other subjects. MBZUAI leads the work with university and industry partners.

Openness

2 medium confidence
2.0
license
cc-by-nc-4.0 (Hub card metadata; the GitHub README License section says CC BY-NC-SA 4.0)
access
public (ungated on the Hub; also in the GitHub data folder)
dataset_card
present (card and README describe sources, construction and statistics)

Anyone can download the questions and the evaluation code, and the card and paper explain how they were collected. The noncommercial license keeps it out of commercial evaluation pipelines, and the Hub and GitHub copies disagree on whether a share-alike condition applies.

Adoption

2 high confidence
2.0

Hugging Face downloads of the Hub copy. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations. It misses copies taken from the GitHub data folder.

Capability

5 medium confidence
5.0

In its one target language, ArabicMMLU's deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

Verified 2026-09-24