AI Potluck
Model components / Benchmark / eval datasets

CRUXEval

Meta

800 Python function, input and output triples evaluating code reasoning and execution through two paired tasks: predicting the input given a function and its output, and predicting the output given a function and its input. Bootstrapped by Meta FAIR using Code Llama 34B and filtered to short, low-compute problems, it was published at ICML 2024 and spawned a multilingual community extension.

The repository was archived read-only by its owner on 18 September 2025, though the benchmark remains in comparative use. Verified 2026-08-13 via the facebookresearch/cruxeval repository.

Openness

5 high confidence
5.0
data
open(800 samples public on GitHub + HF)
license
MIT(permissive, redistributable)
datasheet
yes(README + arXiv paper documenting construction)
access
public(downloadable, fully released, not held-out)

Fully open public benchmark dataset under MIT with documented construction methodology. The repository is archived by its owner and read-only, which changes maintenance rather than openness - an archived public repo is still downloadable and still MIT.

  • https://github.com/facebookresearch/cruxeval recorded 2026-08-13

    the repository's license tab reads "MIT license" and the README ends "CRUXEval is MIT licensed, as found in the LICENSE file"; the README describes "a benchmark of 800 Python functions and input-output pairs" with the CRUXEval-I and CRUXEval-O tasks and the sampling filter; the repo is public (banner: archived by the owner on Sep 18, 2025, read-only) so the samples still download

Adoption

2 high confidence
2.0

The declared dataset artifact reads 8,063 downloads in the trailing 30 days, which puts it in the 1K-10K band, level 2 on the dataset scale; where a real download signal exists it governs over reported traction. The benchmark repository is no longer maintained - its owner archived it on 2025-09-18 and it is read-only. Citation as a research and evaluation standard is real enough, but it is not a usage count and cannot hold up a level the measured signal contradicts.

Capability

not assessed

Capability is not a meaningful axis for an eval dataset, so it is left unscored. The repository publishes a static set of 800 function/input/output triples, with no performance or feature claim for the capability axis to read.

  • https://github.com/facebookresearch/cruxeval recorded 2026-08-13

    a static 800-sample corpus of Python functions with input and output pairs; the README reports how models score on it, never a capability of the corpus itself

Verified 2026-08-13