Back to Gap Map Model components / Benchmark / eval datasets

CMMLU

Haonan Li
restricted / Overall score: 3.0

CMMLU tests language models on 11,528 multiple-choice questions in Mandarin Chinese across 67 subjects, from elementary school to professional level, in the sciences, humanities and social sciences. Many subjects, such as Chinese law, driving rules and food culture, have answers specific to China. Each subject has a five-question development set and a test set of more than a hundred questions. Researchers at MBZUAI, Shanghai Jiao Tong University, Microsoft Research Asia and other labs built it.

Openness

2 high confidence
2.0
license
cc-by-nc-sa-4.0 (the card's license section and the repository README; the card metadata tags cc-by-nc-4.0, also non-commercial)
access
public (ungated on the Hub)
dataset_card
present (card describes the subjects, splits and question format, and links the paper)

The questions download freely, but the license forbids commercial use. The card's metadata and its license section name different Creative Commons variants, and both are non-commercial.

Adoption

3 high confidence
3.0

Hugging Face downloads of the Hub repository over the trailing month. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.

Capability

not assessed

A benchmark is not scored for capability, so that score is left blank. CMMLU tests Mandarin Chinese, a language with a large written web, so it sits with general evaluation sets rather than with sets built for an underrepresented language.

  • https://arxiv.org/abs/2306.09212 recorded 2026-11-12

    Abstract: "This paper aims to address this issue for Mandarin Chinese in the form of CMMLU, a comprehensive Chinese benchmark".

  • https://arxiv.org/pdf/2306.09212 recorded 2026-11-12

    "CMMLU contains 11,528 questions across 67 subjects. Each subject has at least 105 questions, which we split into a few-shot development set with 5 questions, and a test set"; authors from MBZUAI, LibrAI, Shanghai Jiao Tong University, Microsoft Research Asia and The University of Melbourne.

Verified 2026-11-12