Princeton NLP / OpenAI

license

MIT(SWE-bench repository states MIT over its code and data

redistributability

full(downloadable parquet, ungated)

datasheet

yes(dataset card + arXiv:2310.06770)

held-out

no hidden split in Verified itself(the public 500-instance set is fully released

data-source

real GitHub PR/issue pairs

SWE-bench Verified is ungated and downloadable, with the gold patches and tests included; the Hugging Face card itself states no license, but the canonical SWE-bench repository's citation-and-license section names MIT, and a repository's own license governs when the two disagree. That makes it fully open, unlike the broader SWE-bench variants, which can carry held-out splits.

• https://github.com/SWE-bench/SWE-bench/blob/main/LICENSE recorded 2026-06-04

  MIT License for the SWE-bench harness

• https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified recorded 2026-09-29

  The dataset card reads "SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality" and documents the FAIL_TO_PASS and PASS_TO_PASS fields; searched the page for a license and found none.

• https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified/blob/main/README.md recorded 2026-09-29

  The rendered README view shows the card text "SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality"; searched the body for a license key: none.

• https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified/raw/main/README.md recorded 2026-09-29

  The raw card holds "SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality" and its front matter has no license key; searched for license, none.

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/LICENSE recorded 2026-09-29

  The LICENSE file opens "MIT License" and reads "Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan" followed by the standard MIT grant "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell".

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-09-29

  The README opens with "Code and data for the following works:" and its "Citation & license" section reads "MIT license. Check `LICENSE.md`.", so the repository states MIT over code and data; it still lists no license for the Hugging Face copy itself.

Adoption is measured by Hugging Face downloads across the declared SWE-bench Verified artifacts. GPT-5, Claude Opus, DeepSeek, Qwen and Kimi all report SWE-bench Verified as their headline coding metric, and a live, heavily submitted leaderboard runs at swebench.com.

• https://huggingface.co/api/datasets/princeton-nlp/SWE-bench_Verified recorded 2026-08-13

  413,323 downloads in the trailing 30 days for princeton-nlp/SWE-bench_Verified

A dataset is not 'capable', so capability is left unscored. SWE-bench Verified is notably contamination-resistant and discriminating for coding agents, since it runs against real repository state and executable tests, but that quality is noted here rather than turned into a number.

• https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified recorded 2026-09-29

  The dataset card describes a corpus: "SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality", with FAIL_TO_PASS and PASS_TO_PASS test fields. Searched the body for SOTA and throughput claims: none.