Back to Gap Map Model components / Benchmark / eval datasets

MLE-bench

gated / Overall score: 2.0

Benchmark measuring how well AI agents perform machine learning engineering, built from 75 Kaggle competitions. OpenAI ships the preparation and grading code plus new train/test splits, and the raw competition data is downloaded from Kaggle under each competition's own rules. A public leaderboard tracks agent submissions, though new submissions are paused as of April 2026.

The LICENSE is MIT over the code with an explicit carve-out excluding the external datasets, and the corpus is downloaded from Kaggle with credentials under each competition's own rules. GitHub-only channels; no first-party HF dataset or PyPI package.

Openness

3 high confidence

3.0

license: per-component(one component per Kaggle competition: the repo LICENSE is MIT over the code only, with the explicit NOTE "This license applies to the code in this repository, but not the external datasets and files that may be downloaded while using this package", so the corpus defers to each competition's own rules)
access: terms-acceptance(data downloaded via the Kaggle API with Kaggle credentials, under Kaggle's per-competition rules
answers: released(the benchmark's own test splits are constructed by the prepare scripts from public training sets, and grading scripts ship in the repo — no split hidden by MLE-bench itself)
datasheet: yes(repository README documents composition (75 competitions), preparation and grading
arXiv: 2410.07095)

MLE-bench's competition data is available only through Kaggle, which requires an account and acceptance of each competition's own terms, so the corpus is gated rather than freely open however permissive its code license is. The repository's MIT license explicitly covers only the harness code, not the datasets or files downloaded through the package, which is also why GitHub reports NOASSERTION.

https://api.github.com/repos/openai/mle-bench recorded 2026-09-01
license spdx NOASSERTION (the classifier choking on the carve-out note), 1,727 stars, homepage openai.com/index/mle-bench
https://raw.githubusercontent.com/openai/mle-bench/main/LICENSE recorded 2026-10-07
MIT License, Copyright (c) 2024 OpenAI, standard text, then: "NOTE: This license applies to the code in this repository, but not the external datasets and files that may be downloaded while using this package." — the carve-out that makes the corpus deferred and explains GitHub's NOASSERTION
https://raw.githubusercontent.com/openai/mle-bench/main/README.md recorded 2026-10-07
dataset is 75 Kaggle competitions; prepare scripts split public training sets into new train/test; raw data downloaded with the Kaggle API requiring kaggle.json credentials; grading scripts per competition; leaderboard paused for new submissions 2026-04-24

Adoption

2 medium confidence

2.0

GitHub stars are the only adoption signal available for MLE-bench: there is no Hugging Face dataset or PyPI package, and the underlying competition data comes from Kaggle, which publishes no per-benchmark download figure. A star is not a use, and no vendor-published usage figure exists to check it against.

https://api.github.com/repos/openai/mle-bench recorded 2026-09-01
stargazers_count 1727

Capability

not assessed

A dataset is not 'capable', so capability is left unscored, mirroring the category pattern (gsm8k).

https://raw.githubusercontent.com/openai/mle-bench/main/README.md recorded 2026-10-07
a benchmark definition plus leaderboard of OTHER systems' scores; no capability claim about the artifact itself

Verified 2026-10-07, except some scores, still awaiting re-confirmation