Back to Gap Map Organization

Haonan Li

individual

Scores

1 product on the map — 1 closed.

CMMLU

Openness

2 high confidence
2.0
license
cc-by-nc-sa-4.0 (the card's license section and the repository README; the card metadata tags cc-by-nc-4.0, also non-commercial)
access
public (ungated on the Hub)
dataset_card
present (card describes the subjects, splits and question format, and links the paper)

The questions download freely, but the license forbids commercial use. The card's metadata and its license section name different Creative Commons variants, and both are non-commercial.

Adoption

3 high confidence
3.0

Hugging Face downloads of the Hub repository over the trailing month. A download count does not say who downloaded, and evaluation harnesses can fetch a benchmark repeatedly, so the figure is a rough measure of use, not a count of users or organizations.

Capability

not assessed

A benchmark is not scored for capability, so that score is left blank. CMMLU tests Mandarin Chinese, a language with a large written web, so it sits with general evaluation sets rather than with sets built for an underrepresented language.

  • https://arxiv.org/abs/2306.09212 recorded 2026-11-12

    Abstract: "This paper aims to address this issue for Mandarin Chinese in the form of CMMLU, a comprehensive Chinese benchmark".

  • https://arxiv.org/pdf/2306.09212 recorded 2026-11-12

    "CMMLU contains 11,528 questions across 67 subjects. Each subject has at least 105 questions, which we split into a few-shot development set with 5 questions, and a test set"; authors from MBZUAI, LibrAI, Shanghai Jiao Tong University, Microsoft Research Asia and The University of Melbourne.