Back to Gap Map Model components / Benchmark / eval datasets

HumanEval

open / Overall score: 4.0(strong)

164 hand-written Python programming problems, each with a function signature, docstring and unit tests, introduced in the Codex paper in 2021. It was the original code-generation benchmark and most code-model papers still report pass@k on it, despite its small size and Python-only scope.

Openness

5 high confidence

5.0

license: MIT(OSI-style permissive, fully redistributable)
redistributability: full(downloadable parquet, no gating)
datasheet: yes(dataset card)
size: 164 problems/test split
contamination: hand-written to avoid GitHub-dump overlap (per design)

Fully open: MIT license, ungated, downloadable, with a dataset card. Textbook open-data case for this category.

https://huggingface.co/datasets/openai/openai_humaneval recorded 2026-09-29
The dataset page tags the corpus "license:mit", reports "gated":false, and renders a card titled "Dataset Card for OpenAI HumanEval" with a "num_examples":164 test split; searched the body for accuracy and SOTA claims: none.

Adoption

4 high confidence

4.0

Adoption is measured through Hugging Face downloads of openai/openai_humaneval. HumanEval is the foundational code-generation benchmark — the pass@k metric originated here — and it is still reported in most code-model releases, though this category counts dataset downloads on a far smaller scale than it does for software or models.

https://huggingface.co/api/datasets/openai/openai_humaneval recorded 2026-08-12
279,859 downloads in the trailing 30 days for openai/openai_humaneval

Capability

not assessed

A dataset is not 'capable', so capability is left unscored. It is worth noting that HumanEval is now largely saturated and contaminated as a discriminator, but that is a quality caveat rather than a capability score.

https://huggingface.co/datasets/openai/openai_humaneval recorded 2026-09-29
The dataset page describes a static corpus: "The HumanEval dataset released by OpenAI includes 164 programming problems" with columns "canonical_solution" and "entry_point"; searched the body for throughput and SOTA claims: none.

Verified 2026-09-27