Model components / Evaluation code

SWE-bench

Docker-based evaluation harness that benchmarks coding agents on 2,294 real GitHub issues from 12 popular Python repositories, producing pass-rate scores against held-out tests. Picked because it is the de-facto industry standard for measuring software-engineering agent capability. Every frontier lab (Anthropic, OpenAI, Google) reports SWE-bench Verified scores. ~5k stars on GitHub and Epoch AI maintains an authoritative leaderboard. Note: OpenAI's April 2026 audit flagged that 59.4% of the hardest unsolved problems in SWE-bench Verified had flawed test cases; they now prefer the harder SWE-bench Pro and SWE-bench Bash variants for frontier evaluation.

SWE-bench evaluation harness. PyPI `swebench` v4.1.0 (Sep 11 2025); repo migrated to github.com/SWE-bench/SWE-bench, last headline feature SWE-bench Multimodal (Jan 2025). Confirmed live June 2026. Scoring the HARNESS (the software that runs models against GitHub-issue tasks); the underlying dataset belongs in benchmark_eval_data.

Openness

5 high confidence

5.0

license: MIT(OSI)
source: public(GitHub)
core-gated: ungated

MIT-licensed harness, full source public; OSI -> open_source.

https://pypi.org/project/swebench/ recorded 2026-06-04
official SWE-bench package, MIT license, v4.1.0
https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04
MIT license, public harness source

Adoption

4 medium confidence

4.0

SWE-bench Verified is the de-facto-standard agentic-coding eval cited in essentially every 2025-26 frontier release report (GPT-5 74.9%, DeepSeek-V4-Pro 80.6%, Qwen3.6, Kimi K2.6) -- the top adoption signal for a harness per recipe. pypistats reports ~31.5M swebench downloads/mo but that figure is almost certainly CI/benchmark-runner-inflated and not human installs, so used as corroboration only, not the headline; level set on standard-status, capped at 4.

https://pypistats.org/packages/swebench recorded 2026-06-04
~31.5M monthly PyPI downloads (likely CI-inflated; corroboration only)
https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04
ICLR 2024 oral; harness used across model evaluations

Capability

5 high confidence

5.0

Frontier-definer within evaluation_code for agentic/coding eval -- the reference harness for the benchmark every lab reports against.

https://github.com/princeton-nlp/SWE-bench recorded 2026-06-04
harness runs models on real GitHub issues with test-based verification; multimodal split

Unchanged since 2026-07-30 (last edited, not re-checked)