Model components / Benchmark / eval datasets

SWE-bench Verified

500 human-verified instances drawn from the broader SWE-bench dataset, each a real GitHub issue paired with a test-verified solution patch. Created through collaboration between Princeton NLP and OpenAI to reduce noise in the original 2,294-instance set. The gold-standard benchmark for evaluating coding agents on realistic software engineering tasks. Leaderboard results drive agent development across the industry. As of April 2026, OpenAI's internal audit found 59.4% of the hardest unsolved problems had flawed test cases and that frontier models can reproduce gold patches verbatim. OpenAI now prefers SWE-bench Pro for headline coding evaluations.

SWE-bench Verified (princeton-nlp/SWE-bench_Verified), 500 human-validated GitHub issue-resolution task instances (~2.1 MB) curated by OpenAI from the SWE-bench test set. Leaderboard at swebench.com. Verified live on HF June 2026 (~914k downloads last month).

Openness

4 medium confidence

4.0

license: undeclared on HF dataset card (underlying SWE-bench harness repo is MIT)
redistributability: full(downloadable parquet, ungated)
datasheet: yes(dataset card + arXiv:2310.06770)
held-out: no hidden split in Verified itself (the public 500-instance set is fully released
data-source: real GitHub PR/issue pairs

Effectively open (ungated, downloadable, full gold patches and tests, datasheet present) but the HF dataset card declares NO explicit license field, so the data-license class is inferred from the MIT harness repo and public redistribution rather than a stated data license. Scored 4 not 5 on that ambiguity. (Distinct from SWE-bench's broader/hidden variants which can carry held-out splits.)

https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified recorded 2026-06-04
500 instances, ~2.1 MB, not gated, dataset card present, gold patch + FAIL_TO_PASS/PASS_TO_PASS tests included, leaderboard swebench.com
https://github.com/SWE-bench/SWE-bench/blob/main/LICENSE recorded 2026-06-04
MIT License for the SWE-bench harness
https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified/blob/main/README.md recorded 2026-06-04
dataset card metadata has no license field declared

Adoption

4 high confidence

4.0

The de-facto standard agentic-coding benchmark of 2025-2026: cited as the headline coding metric in nearly every frontier model release (GPT-5, Claude Opus, DeepSeek, Qwen, Kimi all report SWE-bench Verified). ~914k HF downloads/month plus a live, heavily-submitted leaderboard at swebench.com. Strongest 'citation-as-standard' signal in the batch.

https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified recorded 2026-06-04
~914k downloads last month; leaderboard at swebench.com

Capability

not assessed

Dataset, not 'capable'. Notably high contamination-resistance and discriminating power for coding agents (real repo state, executable tests), but per recipe capability is n/a; the quality proxy is noted, not numerically scored.

Unchanged since 2026-06-24 (last edited, not re-checked)