Back to Gap Map Model components / Language-specific evaluation sets

TurkishMMLU

Arda Yüksel
restricted / Overall score: 3.8

TurkishMMLU tests language models on multiple-choice questions from Turkish high-school curricula, covering nine subjects in natural sciences, mathematics, Turkish language and literature, and social sciences and humanities. Curriculum experts wrote the more than 10,000 questions, and each carries a correctness ratio as a difficulty signal. Arda Yüksel, Abdullatif Köksal and colleagues publish it with evaluation code.

The Hub repository holds dev and test splits of 1,845 rows; the full question set is available by emailing the authors, as the README states.

Openness

2 medium confidence
2.0
license
not-clearly-stated-on-card (no license in the Hub card metadata or API tags; the GitHub repository has no LICENSE file and its README names none)
access
public (the Hub subset is ungated; the full question set is by email request to the authors)
dataset_card
present (card gives the abstract, subjects and evaluation results)

Only a subset of the questions is openly downloadable, and the rest must be requested from the authors by email. No license is stated anywhere, so reuse terms for either part are unknown.

Adoption

1 high confidence
1.0

Hugging Face downloads of the public subset. Copies of the full set obtained by email are not counted. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.

Capability

5 medium confidence
5.0

In its one target language, TurkishMMLU's deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

Verified 2026-09-24