SWE-bench

license

MIT(OSI)

source

public(GitHub)

core features withheld

no

The harness is MIT-licensed with its full source public, and MIT is an OSI license, so this is open source. The README documents building it from source and running the evaluation locally, lists no paid tier or license key, and states that the Multimodal split is fully open for local evaluation.

• https://api.github.com/repos/SWE-bench/SWE-bench/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for SWE-bench/SWE-bench, and returns the LICENSE.md body as the MIT text.

• https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04

  MIT license, public harness source

• https://pypi.org/project/swebench/ recorded 2026-06-04

  official SWE-bench package, MIT license, v4.1.0

• https://pypi.org/pypi/swebench/json recorded 2026-09-29

  PyPI metadata for swebench lists the license as "MIT License" and carries the classifier "License :: OSI Approved :: MIT License".

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/LICENSE recorded 2026-09-29

  The LICENSE file opens with "MIT License" and grants "Permission is hereby granted, free of charge, to any person obtaining a copy" of the software, with the right to "use, copy, modify, merge, publish, distribute, sublicense, and/or sell" it.

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-09-29

  README carries the MIT badge, closes with "MIT license. Check `LICENSE.md`.", and documents building the harness from source ("pip install -e .") and running Docker evaluation locally with "swebench eval verified". No pricing, enterprise edition, paid tier or license key appears in it (searched for pricing, enterprise, paid, license key). The Multimodal news item states "SWE-bench Multimodal v2 is now fully open source, with 480 tasks available for local evaluation"; sb-cli and Modal are listed only as optional cloud alternatives.

SWE-bench Verified is the de-facto standard agentic-coding evaluation, cited in nearly every 2025-26 frontier release report. Its swebench package also draws heavy PyPI traffic, but that volume is almost certainly CI and benchmark-runner traffic rather than human installs, so citation carries this reading and the download count serves only as corroboration.

• https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04

  ICLR 2024 oral; harness used across model evaluations

• https://pypistats.org/api/packages/swebench/recent recorded 2026-10-08

  last_month = 22,102,583 downloads of the swebench package (last_week 4,466,166, last_day 486,886)

• https://pypistats.org/packages/swebench recorded 2026-06-04

  ~31.5M monthly PyPI downloads (likely CI-inflated; corroboration only)

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-10-08

  README records the ICLR 2024 oral and ICLR 2025 Multimodal papers and the SWE-bench Verified 500-problem split built with OpenAI Preparedness.

The harness applies model-generated patches to real GitHub issues and grades them by running each repository's tests in a Docker container. It documents Verified, Lite and Multimodal splits, and Modal and sb-cli cloud evaluation.

• https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04

  harness runs models on real GitHub issues with test-based verification; multimodal split

• https://huggingface.co/datasets/SWE-bench/SWE-bench/raw/main/README.md recorded 2026-09-29

  Hugging Face dataset card says "The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories" and "Evaluation is performed by unit test verification using post-PR behavior as the reference solution."

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/docs/guides/evaluation.md recorded 2026-09-29

  Evaluation guide states "SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved" in "a containerized Docker environment", and documents Modal and sb-cli cloud evaluation.

• https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/README.md recorded 2026-09-29

  README documents Verified, Lite and Multimodal datasets and the eval command: "swebench eval multimodal --gold", "swebench infer verified -m gpt-5 -o preds -w 8 # generate predictions with mini-SWE-agent", and "Running SWE-bench evaluation on [Modal](https://modal.com/)" as a cloud option.