Back to Gap Map Model components / Language-specific evaluation sets

Khayyam Challenge

Raia Center
restricted / Overall score: 3.8

The Khayyam Challenge, also called PersianMMLU, tests language models on 20,192 four-choice questions in Persian from 38 tasks drawn from Persian examinations, from lower primary to upper secondary school. The questions are original rather than translated and carry metadata such as human response rates, difficulty ratings, and descriptive answers. The RAIA Center publishes it.

The Hub README is empty; the description follows the paper and the gating terms.

Openness

2 high confidence
2.0
license
cc-by-nd-4.0 (card metadata; the gating terms add non-commercial, academic-research-only use)
access
manual (manual approval after a form sharing contact information and accepting the terms)
dataset_card
no (README.md exists but is empty)

Access needs a form and manual approval, and the terms allow only non-commercial academic research and forbid derivative works, including new benchmarks built from the questions. The repository has no card, so its contents are described only in the paper. The no-derivatives term alone would keep it from being open data, since a model trained on the questions is arguably a derivative.

Adoption

1 high confidence
1.0

Hugging Face downloads of the gated repository over the trailing month. Access requires approval, but the download count does not identify unique users or organizations, and repeated downloads are possible. Copies used by leaderboards are not counted.

Capability

5 medium confidence
5.0

In its one target language, Khayyam Challenge's deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2404.06644 recorded 2026-11-08

    Abstract: "Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations"; "its use of original, non-translated data tailored for Persian speakers".

  • https://arxiv.org/html/2404.06644 recorded 2026-11-08

    Settings: "we benchmarked a standardized prompt-0" "template on the entire dataset", with two other templates and chain-of-thought on subsets of 1000; the dataset is "originally constructed in Persian"; questions come from "Iran’s national university entrance" and "Kanoon Farhangi Amoozesh (Cultural Educational Center), wherein questions are not only verified and validated by experts in each subject". No train, dev or test split is described.

  • https://huggingface.co/datasets/raia-center/khayyam-challenge recorded 2026-11-08

    Hub page: "Space using raia-center/khayyam-challenge 1: ParsBench/leaderboard".

Verified 2026-09-24