Back to Gap Map Model components / Language-specific evaluation sets

KazMMLU

Mohamed bin Zayed University of Artificial Intelligence
restricted / Overall score: 3.8

KazMMLU tests language models on 23,000 multiple-choice questions in Kazakh and Russian, 10,969 and 12,031 respectively, reflecting Kazakhstan's bilingual education system. The questions cover STEM, humanities, and social sciences in high-school and university material, sourced from authentic educational materials and manually validated by native speakers and educators. MBZUAI researchers built it.

The Hub card is a stub from before the repository was renamed and does not describe the release; the description follows the paper.

Openness

2 high confidence
2.0
license
cc-by-nc-4.0 (card metadata and API tag)
access
public (ungated on the Hub)
dataset_card
partial (card lists splits and columns but is titled with another repository name and truncates its subset list)

The questions download freely under a noncommercial license, which rules out commercial evaluation use. The card covers only the file layout, so how the questions were sourced is documented in the paper alone.

Adoption

1 high confidence
1.0

Hugging Face downloads of the Hub repository. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.

Capability

5 medium confidence
5.0

In the typical language KazMMLU targets, its deepest test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2502.12829 recorded 2026-11-08

    Abstract: "KazMMLU, the first MMLU-style dataset specifically designed for Kazakh language. KazMMLU comprises 23,000 questions ... sourced from authentic educational materials and manually validated by native speakers and educators".

  • https://arxiv.org/pdf/2502.12829 recorded 2026-11-08

    "The dataset includes 10,969 Kazakh questions and 12,031 Russian questions"; "KazMMLU is sourced from authentic educational materials, including national exams, textbooks, and professional certification repositories"; "the question and answer choices remain in their original language (Kazakh or Russian)"; "every question included in KazMMLU undergoes manual validation"

  • https://datasets-server.huggingface.co/size?dataset=MBZUAI/KazMMLU recorded 2026-11-08

    the Hub's row counts per subject configuration, each named for its language, such as "Biology (High School in kaz)" and "Biology (High School in rus)": test 9,870 rows across the 12 Kazakh configurations and 13,019 across the 25 Russian ones; dev 3 rows in each configuration (36 Kazakh, 75 Russian); 23,000 rows in all

  • https://huggingface.co/datasets/MBZUAI/KazMMLU recorded 2026-11-08

    Hub page: "Number of rows: 23,000"; no models or Spaces listed as using it.

Verified 2026-09-24