Back to Gap Map Model components / Benchmark / eval datasets

EvalPlus

open / Overall score: 3.0

Extensions of HumanEval and MBPP that add roughly 80 times and 35 times more test cases through automated mutation, exposing models that pass weak tests while emitting buggy code. Published at NeurIPS 2023, with an EvalPerf companion for efficiency, and integrated with the bigcode evaluation harness.

The original benchmarks are saturated above 90 percent, which is why the Plus variants are used to separate frontier models.

Openness

5 high confidence

5.0

data: downloadable(HF parquet, evalplus/humanevalplus + mbppplus)
license: Apache-2.0(OSI/open, on both framework and datasets)
datasheet: present(GitHub README documents HumanEval+ and MBPP+ (test counts, task fixes, paper link)
redistributable: yes
framework-code: open(Apache-2.0)

The HumanEval+ and MBPP+ datasets and the EvalPlus evaluation framework are all released under Apache-2.0, and the data downloads from Hugging Face without a gate. The Hub dataset cards are empty, but the GitHub README and the accompanying paper document how the test suites were extended and which tasks were fixed.

https://github.com/evalplus/evalplus recorded 2026-08-13
the repository's license tab still reads Apache-2.0, and the README still describes the HumanEval+ and MBPP+ variants and the bigcode-evaluation-harness integration
https://huggingface.co/api/datasets/evalplus/humanevalplus recorded 2026-09-29
The Hub API record lists the tag "license:apache-2.0", reports "gated":false, and lists "data/test-00000-of-00001-5973903632b82d40.parquet" among its siblings.
https://huggingface.co/datasets/evalplus/humanevalplus recorded 2026-09-29
The page carries "license:apache-2.0" and lists "data/test-00000-of-00001-5973903632b82d40.parquet", while its card body reads "README.md exists but content is empty".
https://huggingface.co/datasets/evalplus/mbppplus recorded 2026-09-29
The MBPP+ page carries "license:apache-2.0" and lists "data/test-00000-of-00001-d5781c9c51e02795.parquet", while its card body reads "README.md exists but content is empty".
https://raw.githubusercontent.com/evalplus/evalplus/master/LICENSE recorded 2026-09-29
The repository LICENSE file opens with "Apache License" and "Version 2.0, January 2004", the framework code license.
https://raw.githubusercontent.com/evalplus/evalplus/master/README.md recorded 2026-09-29
The README describes HumanEval+ with "80x more tests than the original HumanEval!" and MBPP+ with "35x more tests than the original MBPP!", and logs a dataset revision as "removing some broken tasks (399 -> 378 tasks)".

Adoption

3 medium confidence

3.0

Adoption is measured by Hugging Face downloads summed across the two companion datasets, evalplus/humanevalplus and evalplus/mbppplus. Both are contamination-hardened versions of HumanEval and MBPP, reported adopted by Meta and other labs for coding evaluation.

https://huggingface.co/api/datasets/evalplus/humanevalplus recorded 2026-08-12
27,086 downloads in the trailing 30 days for evalplus/humanevalplus
https://huggingface.co/api/datasets/evalplus/mbppplus recorded 2026-08-12
23,995 downloads in the trailing 30 days for evalplus/mbppplus

Capability

not assessed

A dataset is not capability-scored, so capability is left unscored. Its distinguishing quality - far denser test suites that catch overfitting the base HumanEval and MBPP suites miss - is noted here rather than turned into a number.

https://huggingface.co/datasets/evalplus/humanevalplus recorded 2026-09-29
Dataset page for a static problem corpus, with parquet columns such as "Column 'prompt' from the Hugging Face parquet file."; searched the body for leaderboard, accuracy, SOTA and pass@1: none, no performance, throughput or feature claim.

Verified 2026-09-29, except some scores, still awaiting re-confirmation