Back to Gap Map Model components / Benchmark / eval datasets

Humanity's Last Exam (HLE)

gated / Overall score: 3.0

Multimodal frontier-knowledge benchmark from the Center for AI Safety and Scale AI, built with nearly a thousand subject-matter experts across more than 500 institutions. It holds 2,500 expert-written closed-ended questions, multiple-choice and short answer with text and images, across more than a hundred subjects, each with an unambiguous answer that is not quickly findable online.

A separate private held-out set detects overfitting, and it is described on the project site rather than the dataset card.

Openness

2 medium confidence

2.0

license: MIT(the public 2,500-question set)
questions+answers: public(2500-Q)
access: HF-gated(accept-conditions,no-redistribution,canary)
holdout: private

The public set is MIT-licensed and ships with its answers, but access is gated and a private held-out set exists, which is what decides the score. The private holdout is not mentioned on the Hugging Face card, so lastexam.ai is cited for it: it reads "We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting."

https://huggingface.co/datasets/cais/hle recorded 2026-09-29
The Hugging Face dataset card carries the `license:mit` tag and says "You need to agree to share your contact information to access this dataset" with the repo state showing "gated":"auto", and it documents a canary string that is "a superset of the BIG-bench canary string"; it makes no statement about a private held-out set.
https://lastexam.ai recorded 2026-09-29
The project site says "We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting" and "HLE has been finalized with 2,500 questions."

Adoption

3 high confidence

3.0

Adoption is measured by Hugging Face downloads of cais/hle. HLE is a frontier-difficulty benchmark whose download volume has stayed flat and steady rather than growing or declining.

https://huggingface.co/api/datasets/cais/hle recorded 2026-08-12
34,715 downloads in the trailing 30 days for cais/hle

Capability

not assessed

A dataset is not 'capable' the way software or a model is, so capability is left unscored. HLE is a successor to MMLU and MATH built to resist saturation: 2,500 expert-written questions across more than 100 subjects, multimodal, with a private held-out set, and harder and broader than GPQA, its nearest narrower peer.

https://lastexam.ai recorded 2026-09-29
The project site describes the corpus as a question set: "HLE has been finalized with 2,500 questions." and says "We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting"; searched the body for a runnable tool or API offering: none.

Verified 2026-09-27