Model components / Benchmark / eval datasets

APPS

Automated Programming Progress Standard, 10,000 coding problems at three difficulty levels (introductory, interview, competition) drawn from open-access coding challenge sites. Published at NeurIPS 2021. One of the first large-scale code benchmarks to test models across a realistic difficulty spectrum, bridging the gap between simple function-completion tasks and full competitive programming. Still used as a reference dataset for training and evaluation.

APPS (Hendrycks et al., NeurIPS 2021, arXiv 2105.09938): 10,000 coding problems (5,000 train / 5,000 test) with 131,777 test cases, Introductory/Interview/Competition difficulty, Python solutions. HF dataset (codeparrot/apps) verified live June 2026 (13,834 monthly downloads, MIT).

Openness

5 high confidence

5.0

data: downloadable(HF parquet, codeparrot/apps)
license: MIT(OSI/open)
datasheet: present(card documents 10k problems, 131,777 tests, difficulty levels, schema)
redistributable: yes
paper: public(NeurIPS 2021)

Clean open data: MIT, downloadable, documented datasheet, public paper: full open class.

https://huggingface.co/datasets/codeparrot/apps recorded 2026-06-04
MIT license, 10,000 problems, 131,777 test cases, 13,834 monthly downloads

Adoption

3 medium confidence

3.0

13,834 monthly HF downloads (primary). A well-established code-generation benchmark cited widely since 2021, but by 2026 partly superseded by harder/contamination-free suites (LiveCodeBench, SWE-bench) for frontier evaluation. Level 3 on download volume + sustained-but-not-headline citation; honest read is a foundational-but-aging standard.

https://huggingface.co/datasets/codeparrot/apps recorded 2026-06-04
13,834 monthly downloads

Capability

not assessed

Dataset, not capability-scored.

Unchanged since 2026-06-24 (last edited, not re-checked)