Back to Gap Map Organization

Raia Center

lab

Scores

1 product on the map — 1 closed.

Khayyam Challenge

Openness

2 high confidence
2.0
license
cc-by-nd-4.0 (card metadata; the gating terms add non-commercial, academic-research-only use)
access
manual (manual approval after a form sharing contact information and accepting the terms)
dataset_card
no (README.md exists but is empty)

Access needs a form and manual approval, and the terms allow only non-commercial academic research and forbid derivative works, including new benchmarks built from the questions. The repository has no card, so its contents are described only in the paper. The no-derivatives term alone would keep it from being open data, since a model trained on the questions is arguably a derivative.

Adoption

1 high confidence
1.0

Hugging Face downloads of the gated repository over the trailing month. Access requires approval, but the download count does not identify unique users or organizations, and repeated downloads are possible. Copies used by leaderboards are not counted.

Capability

5 medium confidence
5.0

In its one target language, Khayyam Challenge's deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2404.06644 recorded 2026-11-08

    Abstract: "Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,192 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations"; "its use of original, non-translated data tailored for Persian speakers".

  • https://arxiv.org/html/2404.06644 recorded 2026-11-08

    Settings: "we benchmarked a standardized prompt-0" "template on the entire dataset", with two other templates and chain-of-thought on subsets of 1000; the dataset is "originally constructed in Persian"; questions come from "Iran’s national university entrance" and "Kanoon Farhangi Amoozesh (Cultural Educational Center), wherein questions are not only verified and validated by experts in each subject". No train, dev or test split is described.

  • https://huggingface.co/datasets/raia-center/khayyam-challenge recorded 2026-11-08

    Hub page: "Space using raia-center/khayyam-challenge 1: ParsBench/leaderboard".