Model components / Benchmark / eval datasets

MMLU

Massive Multitask Language Understanding: a 15,908-question benchmark spanning 57 subjects, designed to measure broad knowledge and reasoning across humanities, social sciences, STEM, and professional domains. It became one of the default headline evals for frontier models.

MMLU (Massive Multitask Language Understanding; Hendrycks et al., ICLR 2021, arXiv 2009.03300): 57 subjects, ~14,042 test questions (231,400 total rows incl. auxiliary_train), 4-option multiple choice. HF dataset (cais/mmlu) verified live June 2026 (541,723 monthly downloads, MIT). Note: now largely saturated (top models 88-94%) and largely superseded by MMLU-Pro for frontier differentiation.

Openness

5 high confidence

5.0

data: downloadable(HF parquet, cais/mmlu)
license: MIT(OSI/open)
datasheet: present(card documents 57 subjects, splits dev/val/test, schema)
redistributable: yes
paper: public(ICLR 2021)

Clean open data: MIT, downloadable, documented datasheet, public paper: full open class.

https://huggingface.co/datasets/cais/mmlu recorded 2026-06-04
MIT license, 57 subjects, 231,400 rows / 14,042 test, 541,723 monthly downloads, documented splits

Adoption

5 high confidence

5.0

541,723 monthly HF downloads (primary), highest in this batch. MMLU is the canonical general-knowledge LLM benchmark, reported in essentially every frontier model release (OpenAI, Anthropic, Google, Meta, DeepSeek, Qwen) since 2021, the strongest possible release-report standard signal. Even post-saturation it remains a universal reference. Level 5 on download volume + ubiquitous release-report citation.

https://huggingface.co/datasets/cais/mmlu recorded 2026-06-04
541,723 monthly downloads
https://www.lxt.ai/blog/llm-benchmarks/ recorded 2026-06-04
MMLU listed among the standard LLM benchmarks (corroborating, not headline)

Capability

not assessed

Dataset, not capability-scored. Saturation/contamination (top models 88-94%, superseded by MMLU-Pro for differentiation) is a quality caveat noted in version_note, not an openness or capability deduction.

Unchanged since 2026-06-24 (last edited, not re-checked)