SWE-bench

license

MIT(OSI)

source

public(GitHub)

core-gated

ungated

The harness is MIT-licensed and its full source is public, and MIT is an OSI license, so this is open source. The product declares the SWE-bench/SWE-bench repository, with the cited princeton-nlp path redirecting there, and the README ships the whole harness with no paid tier.

• https://pypi.org/project/swebench/ recorded 2026-06-04

  official SWE-bench package, MIT license, v4.1.0

• https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04

  MIT license, public harness source

• https://api.github.com/repos/SWE-bench/SWE-bench/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for SWE-bench/SWE-bench, and returns the LICENSE.md body as the MIT text.

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-08-13

  README carries the MIT badge, closes with 'MIT license. Check LICENSE.md', and documents building the harness from source and running containerized Docker evaluation on the reader's own machine. No pricing, enterprise edition, paid tier or license key appears anywhere in it. The only held-back piece is the SWE-bench Multimodal test split, whose grading runs through the free sb-cli service - a contamination control on one split, not a commercial gate on the harness.

SWE-bench Verified is the de-facto-standard agentic-coding eval, cited in essentially every 2025-26 frontier release report (GPT-5 74.9%, DeepSeek-V4-Pro 80.6%, Qwen3.6, Kimi K2.6), which is the strongest adoption signal a harness can have. The swebench package also draws heavy PyPI traffic - 45,994,790 downloads in the trailing 30 days, against ~31.5M at an earlier reading - but that volume is almost certainly CI and benchmark-runner traffic rather than human installs, so it counts as corroboration only. On download volume alone the package would sit a rung higher; the level rests on standard status instead and is capped at 4, because a package pulled by benchmark runners is not a count of people using it.

• https://pypistats.org/packages/swebench recorded 2026-06-04

  ~31.5M monthly PyPI downloads (likely CI-inflated; corroboration only)

• https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04

  ICLR 2024 oral; harness used across model evaluations

• https://pypistats.org/api/packages/swebench/recent recorded 2026-08-13

  last_month = 45,994,790 downloads of the swebench package (last_week 14,827,831, last_day 1,708,049)

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-08-13

  README records the ICLR 2024 oral and ICLR 2025 Multimodal papers and the SWE-bench Verified 500-problem split built with OpenAI Preparedness - the standard-status evidence the level rests on.

The reference harness for the benchmark every lab reports against, and the frontier-definer for agentic and coding evaluation: patch generation against real GitHub issues, containerized test-based grading, and the Verified, Lite and Multimodal splits, all documented in the repo README.

• https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04

  harness runs models on real GitHub issues with test-based verification; multimodal split

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-08-13

  harness generates patches for real GitHub issues and grades them by running the repository's own tests in Docker; SWE-bench Verified, Lite and Multimodal splits documented; Modal cloud execution offered as an alternative backend.