Back to Gap Map Model components / Language-specific evaluation sets

KMMLU

HAE-RAE
restricted / Overall score: 4.4(strong)

KMMLU tests language models on 35,030 expert-level multiple-choice questions in Korean across 45 subjects, from the humanities to STEM. The questions come from original Korean exams, mostly professional license tests, rather than translations of English benchmarks. The release adds a 208,522-question training split and 225 few-shot examples. Researchers from Yonsei University, EleutherAI, NAVER Cloud and other labs built it, and the HAE-RAE team publishes it on the Hub.

Openness

2 high confidence
2.0
license
cc-by-nd-4.0 (card metadata and API tag)
access
public (ungated on the Hub)
dataset_card
present (card describes the benchmark, its splits and statistics, and links the paper)

The questions download freely, but the license forbids distributing changed versions, so a translation or a derived benchmark cannot be shared. The card and paper describe how the questions were collected.

Adoption

3 high confidence
3.0

Hugging Face downloads of the Hub repository over the trailing month. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.

Capability

5 medium confidence
5.0

In its one target language, KMMLU's test has more than sixty-four hundred items that count toward depth. An item count says how much a set can test, not how precisely it ranks models.

  • https://arxiv.org/abs/2402.11548 recorded 2026-11-11

    Abstract: "We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM"; "KMMLU is collected from original Korean exams".

  • https://arxiv.org/pdf/2402.11548 recorded 2026-11-11

    "all sourced from original Korean exams without any translated content"; "We predominantly source the questions from Korean License Tests"; "This filtering reduces the dataset by 34% to 243,777 questions".

  • https://datasets-server.huggingface.co/size?dataset=HAERAE-HUB/KMMLU recorded 2026-11-11

    The Hub's row counts over 45 subject configurations, complete ("partial":false): test 35,030, dev 225 and train 208,522 rows, 243,777 in all.

Verified 2026-11-11