Model components / Benchmark / eval datasets

CRUXEval

800 Python (function, input, output) triples evaluating code reasoning, understanding, and execution via two paired tasks: CRUXEval-I (predict input given function + output) and CRUXEval-O (predict output given function + input). Bootstrapped by Meta FAIR using Code Llama 34B and filtered to short, low-compute problems, with a maintained public leaderboard at crux-eval.github.io. Published at ICML 2024, widely reported in code-model papers and model cards as the standard execution-reasoning eval distinct from generation benchmarks like HumanEval/MBPP; spawned a community multilingual extension (CRUXEval-X, 12.66K subjects across 19 languages). Repo archived October 2024 but benchmark remains in active comparative use.

CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation) by Meta/FAIR (Gu, Roziere, Leather, Solar-Lezama, Synnaeve, Wang; arXiv 2401.03065, 2024). 800 Python functions with input/output pairs; two tasks: CRUXEval-I (input prediction) + CRUXEval-O (output prediction). Verified live June 2026 on github.com/facebookresearch/cruxeval.

Openness

5 high confidence

5.0

data: open(800 samples public on GitHub + HF)
license: MIT(permissive, redistributable)
datasheet: yes(README + arXiv paper documenting construction)
access: public(downloadable, fully released, not held-out)

Fully open public benchmark dataset under MIT with documented construction methodology.

https://github.com/facebookresearch/cruxeval recorded 2026-06-04
MIT-licensed 800-sample Python code-reasoning benchmark, publicly downloadable with CRUXEval-I/-O tasks

Adoption

3 medium confidence

3.0

Established code-reasoning benchmark cited in code-model evaluations and research since 2024; from Meta FAIR with a dedicated leaderboard. Solid research/eval-standard traction but a specialist code-execution benchmark, not mass-scale; placed at 3 on reported research/citation traction rather than a measured download figure.

https://github.com/facebookresearch/cruxeval recorded 2026-06-04
maintained Meta FAIR benchmark repo with leaderboard, used as a code reasoning/execution evaluation standard

Capability

not assessed

Capability not a meaningful axis for an eval dataset.

Unchanged since 2026-06-09 (last edited, not re-checked)