BigCodeBench

license

Apache-2.0(OSI)

source

public(GitHub)

core-gated

ungated(no paid tier or license key, harness runs locally with --execution local)

Apache-2.0 and fully public, and Apache-2.0 is an OSI license, so this is open source. The repo is archived and read-only, which freezes the source rather than withdrawing it, so it counts as public and ungated all the same.

• https://github.com/bigcode-project/bigcodebench recorded 2026-06-04

  Apache-2.0 license, public harness, 1140 tasks

• https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md recorded 2026-08-11

  README's evaluation command declares `--execution [e2b|gradio|local]`, so the harness runs end to end on the reader's own machine; the default `gradio` backend is a public Hugging Face evaluator Space the README tells you to clone for speed, and `e2b` is an optional third-party sandbox. Nothing in the README offers a paid tier, a commercial edition, or a license key, and the repo root carries no enterprise directory.

• https://api.github.com/repos/bigcode-project/bigcodebench/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id Apache-2.0 for bigcode-project/bigcodebench and returns the Apache-2.0 text as the LICENSE body.

• https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md recorded 2026-08-13

  The evaluation command still declares `--execution [e2b|gradio|local]`, so the harness runs end to end on the reader's own machine, and nothing in the README offers a paid tier, a commercial edition or a license key.

A narrow, specialist code-generation benchmark. The two declared repos carry 1,573 GitHub stars between them (bigcodebench 519, bigcode-evaluation-harness 1,054), and the Hugging Face leaderboard is active with 163 models evaluated. There is no clean PyPI or download figure for the harness and no published install volume, so stars are the only measure available, and a star count caps a level at 3. Stars are read on their own scale, because a star is not a download: 1,573 bands at 1K-10K stars, level 2, which is where the modest star and leaderboard footprint puts it. The bigcodebench repo is archived and read-only, which is a reason the star count will stop moving rather than a reason to move the level.

• https://github.com/bigcode-project/bigcodebench recorded 2026-06-04

  503 stars; leaderboard with 163 models

• https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard recorded 2026-06-04

  active leaderboard ranking evaluated models

• https://api.github.com/repos/bigcode-project/bigcodebench recorded 2026-08-13

  stargazers_count = 519, forks 75, archived = true, pushed_at 2026-01-03 for bigcode-project/bigcodebench

• https://api.github.com/repos/bigcode-project/bigcode-evaluation-harness recorded 2026-08-13

  stargazers_count = 1,054, forks 261 for bigcode-project/bigcode-evaluation-harness, the second declared repo

A solid, rigorous code-generation eval, but single-benchmark in scope, so it sits below the broad multi-task harnesses on coverage breadth - one rung under OpenCompass, and below lm-eval on the same ground.

• https://github.com/bigcode-project/bigcodebench recorded 2026-06-04

  1140 tasks, Complete/Instruct splits, execution-based eval

• https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md recorded 2026-08-13

  'a new benchmark for code generation with 1140 software-engineering-oriented programming tasks'; Complete and Instruct splits described; generation and execution-based grading run through `--split [complete|instruct]`.