MMLU-Pro
TIGER-Lab'/%3e%3cdefs%3e%3cpattern%20id='pattern0_1240_293949'%20patternUnits='userSpaceOnUse'%20patternTransform='matrix(11.2%200%200%2011.2%203.93617%200)'%20preserveAspectRatio='none'%20viewBox='0%200%2016%2016'%20width='1'%20height='1'%3e%3cg%20id='pattern0_1240_293949_inner'%3e%3crect%20width='16'%20height='16'%20fill='%23f88376'%20mask='url(%23hatch_gaps)'/%3e%3cmask%20id='hatch_gaps'%20maskUnits='userSpaceOnUse'%20x='0'%20y='0'%20width='16'%20height='16'%3e%3crect%20width='16'%20height='16'%20fill='white'/%3e%3crect%20x='6'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='6'%20y='8'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='4'%20y='2'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='4'%20y='10'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='2'%20y='4'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='2'%20y='12'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20y='6'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20y='14'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='14'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='14'%20y='8'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='12'%20y='2'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='12'%20y='10'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='10'%20y='4'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='10'%20y='12'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='8'%20y='6'%20width='2'%20height='2'%20fill='black'/%3e%3crect%20x='8'%20y='14'%20width='2'%20height='2'%20fill='black'/%3e%3c/mask%3e%3c/g%3e%3c/pattern%3e%3c/defs%3e%3c/svg%3e)
Harder successor to MMLU with about 12,032 questions across 14 disciplines, expanding the choice set from four options to ten and removing trivial and noisy items. It is expert-reviewed and contamination-hardened, and ships a 12,000-question test split with a small validation set.
Openness
5 high confidence- license
- MIT(OSI/open,redistributable)
- access
- public(not gated)
- datasheet
- present(comprehensive card+leaderboard)
- splits
- public(test+validation, no hidden held-out)
Fully open: MIT, ungated, redistributable, comprehensive card.
- https://arxiv.org/abs/2406.01574 recorded 2026-09-29
The abstract describes MMLU-Pro as expanding the choice set "from four to ten options" and eliminating trivial and noisy MMLU questions.
- https://huggingface.co/api/datasets/TIGER-Lab/MMLU-Pro recorded 2026-09-29
The Hugging Face API record for the dataset reads `license:mit` in its tags and `"gated":false`, with test and validation splits and the card text "MMLU-Pro dataset is a more robust and challenging massive multi-task understanding dataset".
- https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro recorded 2026-09-29
The dataset page carries the tag `license:mit`, the card opens with "MMLU-Pro dataset is a more robust and challenging massive multi-task understanding dataset", and it describes an "Expert Review:" step with "rigorous scrutiny by a panel of over ten experts"; the leaderboard link renders.
Adoption
4 high confidenceAdoption is measured by Hugging Face downloads of TIGER-Lab/MMLU-Pro. MMLU-Pro is a standard knowledge-and-reasoning benchmark cited across frontier model reports.
- https://huggingface.co/api/datasets/TIGER-Lab/MMLU-Pro recorded 2026-08-12
170,681 downloads in the trailing 30 days for TIGER-Lab/MMLU-Pro
Capability
not assessedA dataset is not 'capable', so capability is left unscored; openness and adoption carry this category.
- https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro recorded 2026-09-29
The page shows a static question corpus with "MMLU-Pro dataset is a more robust and challenging massive multi-task understanding dataset" and a "Downloads last month" panel. Searched the body for throughput, latency, accuracy claims by the dataset itself: none; it is data, not a capable system.
Verified 2026-09-27