CRUXEval

data

open(800 samples public on GitHub + HF)

license

MIT(permissive, redistributable)

datasheet

yes(README + arXiv paper documenting construction)

access

public(downloadable, fully released, not held-out)

Fully open public benchmark dataset under MIT with documented construction methodology. The repository is archived by its owner and read-only, which changes maintenance rather than openness - an archived public repo is still downloadable and still MIT.

• https://arxiv.org/abs/2401.03065 recorded 2026-09-29

  The arXiv abstract documents construction: "a benchmark consisting of 800 Python functions (3-13 lines)" and "we propose a generic recipe for generating our execution benchmark". Searched the body for a leaderboard score, accuracy of the corpus and SOTA claim about the corpus itself: none.

• https://github.com/facebookresearch/cruxeval recorded 2026-08-13

  the repository's license tab reads "MIT license" and the README ends "CRUXEval is MIT licensed, as found in the LICENSE file"; the README describes "a benchmark of 800 Python functions and input-output pairs" with the CRUXEval-I and CRUXEval-O tasks and the sampling filter; the repo is public (banner: archived by the owner on Sep 18, 2025, read-only) so the samples still download

• https://huggingface.co/api/datasets/cruxeval-org/cruxeval recorded 2026-09-29

  The API record carries the tag "license:mit" and lists "test.jsonl" among its files, and describes "a benchmark of 800 Python functions and input-output pairs".

• https://huggingface.co/datasets/cruxeval-org/cruxeval recorded 2026-09-29

  The Hugging Face dataset page tags the license as MIT, is not gated, and its card says "CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation) is a benchmark of 800 Python functions and input-output pairs" and that the data is "MIT licensed"; the test.jsonl file is downloadable.

Adoption is measured by Hugging Face downloads of the cruxeval-org/cruxeval dataset. The GitHub repository has been archived and read-only since 2025-09-18, and although CRUXEval remains a widely cited evaluation standard, a citation is not a download and does not raise what the measured figure shows.

• https://huggingface.co/api/datasets/cruxeval-org/cruxeval recorded 2026-10-07

  12,676 downloads in the trailing 30 days for cruxeval-org/cruxeval

Capability is not a meaningful axis for an eval dataset, so it is left unscored. The repository publishes a static set of 800 function/input/output triples, with no performance or feature claim for the capability axis to read.

• https://github.com/facebookresearch/cruxeval recorded 2026-08-13

  a static 800-sample corpus of Python functions with input and output pairs; the README reports how models score on it, never a capability of the corpus itself

• https://huggingface.co/datasets/cruxeval-org/cruxeval recorded 2026-09-29

  The dataset page describes what the corpus is: "a benchmark of 800 Python functions and input-output pairs" with tasks CRUXEval-I (input prediction) and CRUXEval-O (output prediction). Searched the body for a feature list, throughput and capability claim of the corpus itself: none.