CMMLU
Haonan LiCMMLU tests language models on 11,528 multiple-choice questions in Mandarin Chinese across 67 subjects, from elementary school to professional level, in the sciences, humanities and social sciences. Many subjects, such as Chinese law, driving rules and food culture, have answers specific to China. Each subject has a five-question development set and a test set of more than a hundred questions. Researchers at MBZUAI, Shanghai Jiao Tong University, Microsoft Research Asia and other labs built it.
Openness
2 high confidence- license
- cc-by-nc-sa-4.0 (the card's license section and the repository README; the card metadata tags cc-by-nc-4.0, also non-commercial)
- access
- public (ungated on the Hub)
- dataset_card
- present (card describes the subjects, splits and question format, and links the paper)
The questions download freely, but the license forbids commercial use. The card's metadata and its license section name different Creative Commons variants, and both are non-commercial.
- https://huggingface.co/api/datasets/lmlmcat/cmmlu recorded 2026-11-12
API JSON: "gated":false; tag license:cc-by-nc-4.0; language:zh.
- https://huggingface.co/datasets/lmlmcat/cmmlu/raw/main/README.md recorded 2026-11-12
Card metadata "license: cc-by-nc-4.0"; license section: "The CMMLU dataset is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License"; "We provide development and test dataset for each of 67 subjects".
- https://raw.githubusercontent.com/haonan-li/CMMLU/master/README.md recorded 2026-11-12
Repository README: "The CMMLU dataset is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License".
Adoption
3 high confidenceHugging Face downloads of the Hub repository over the trailing month. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.
- https://huggingface.co/api/datasets/lmlmcat/cmmlu recorded 2026-11-12
"downloads":16879 for lmlmcat/cmmlu.
Capability
not assessedA benchmark is not scored for capability, so that score is left blank. CMMLU tests Mandarin Chinese, a language with a large written web, so it sits with general evaluation sets rather than with sets built for an underrepresented language.
- https://arxiv.org/abs/2306.09212 recorded 2026-11-12
Abstract: "This paper aims to address this issue for Mandarin Chinese in the form of CMMLU, a comprehensive Chinese benchmark".
- https://arxiv.org/pdf/2306.09212 recorded 2026-11-12
"CMMLU contains 11,528 questions across 67 subjects. Each subject has at least 105 questions, which we split into a few-shot development set with 5 questions, and a test set"; authors from MBZUAI, LibrAI, Shanghai Jiao Tong University, Microsoft Research Asia and The University of Melbourne.
Verified 2026-11-12