TurkishMMLU
Arda YükselTurkishMMLU tests language models on multiple-choice questions from Turkish high-school curricula, covering nine subjects in natural sciences, mathematics, Turkish language and literature, and social sciences and humanities. Curriculum experts wrote the more than 10,000 questions, and each carries a correctness ratio as a difficulty signal. Arda Yüksel, Abdullatif Köksal and colleagues publish it with evaluation code.
The Hub repository holds dev and test splits of 1,845 rows; the full question set is available by emailing the authors, as the README states.
Openness
2 medium confidence- license
- not-clearly-stated-on-card (no license in the Hub card metadata or API tags; the GitHub repository has no LICENSE file and its README names none)
- access
- public (the Hub subset is ungated; the full question set is by email request to the authors)
- dataset_card
- present (card gives the abstract, subjects and evaluation results)
Only a subset of the questions is openly downloadable, and the rest must be requested from the authors by email. No license is stated anywhere, so reuse terms for either part are unknown.
- https://huggingface.co/api/datasets/AYueksel/TurkishMMLU recorded 2026-09-24
API JSON: "gated": false; no license tag; tag size_categories:1K<n<10K.
- https://huggingface.co/datasets/AYueksel/TurkishMMLU/raw/main/README.md recorded 2026-09-24
"To access this dataset please send an email to: arda.yueksel@tum.de or akoksal@cis.lmu.de." No license field in the metadata.
- https://raw.githubusercontent.com/ArdaYueksel/TurkishMMLU/main/README.md recorded 2026-09-24
Repository README repeats the email access line and names no license.
Adoption
1 high confidenceHugging Face downloads of the public subset. Copies of the full set obtained by email are not counted. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.
- https://huggingface.co/api/datasets/AYueksel/TurkishMMLU recorded 2026-09-24
436 downloads in the trailing 30 days for AYueksel/TurkishMMLU
Capability
5 medium confidenceIn its one target language, TurkishMMLU's deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.
- https://arxiv.org/abs/2407.12402 recorded 2026-11-08
Abstract: "TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts".
- https://arxiv.org/html/2407.12402 recorded 2026-11-08
"The test set comprises 9,807 multiple-choice questions, with an additional 225 (25 per subject) in the development set."; "The large scale of our test dataset, including 9,807 questions" led to "TurkishMMLU sub with 100 randomly selected questions per subject, totaling 900"; "In all of our evaluations, we use a small subset of TurkishMMLU, TurkishMMLU sub", and for public models "we calculate performance on both TurkishMMLU and TurkishMMLU sub"; "These questions are sourced from a high-quality online learning platform created by the Turkish Ministry of Education"
- https://datasets-server.huggingface.co/size?dataset=AYueksel/TurkishMMLU recorded 2026-11-08
the Hub's split sizes: "config":"All","split":"test" with "num_rows":900; each of the nine subject configs holds test 100 and dev 5. The dataset's total of 1,845 rows adds the All config's 900 test rows to the subject configs' 900 test and 45 dev rows.
- https://huggingface.co/datasets/AYueksel/TurkishMMLU recorded 2026-11-08
Hub page: "Number of rows: 1,845"; "Space using AYueksel/TurkishMMLU 1: afkfatih/turkish-llm-leaderboard".
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/turkishmmlu/config/_turkishmmlu_default_yaml recorded 2026-11-08
the harness's TurkishMMLU template: "dataset_path: AYueksel/TurkishMMLU", "test_split: test", "fewshot_split: dev"
- https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/turkishmmlu/README.md recorded 2026-11-08
"Using these configurations with LM Evaluation Harness, the results of this study are obtained."; repeats "To access this dataset please send an email to"
Verified 2026-09-24