BigCodeBench

license

Apache-2.0(OSI)

source

public(GitHub)

core features withheld

no — no paid tier or license key, harness runs locally with --execution local

Apache-2.0 and fully public, and Apache-2.0 is an OSI license, so this is open source. The repo is archived and read-only, which freezes the source rather than withdrawing it, so it counts as public and ungated all the same.

• https://api.github.com/repos/bigcode-project/bigcodebench/license recorded 2026-09-22

  GitHub's license endpoint reports spdx_id Apache-2.0 for bigcode-project/bigcodebench and returns the Apache-2.0 text as the LICENSE body.

• https://github.com/bigcode-project/bigcodebench recorded 2026-06-04

  Apache-2.0 license, public harness, 1140 tasks

• https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md recorded 2026-09-22

  README's evaluation command declares `--execution [e2b|gradio|local]`, so the harness runs end to end on the reader's own machine; the default `gradio` backend is a public Hugging Face evaluator Space the README tells you to clone for speed, and `e2b` is an optional third-party sandbox. Nothing in the README offers a paid tier, a commercial edition, or a license key, and the repo root carries no enterprise directory.

• https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md recorded 2026-09-22

  The evaluation command still declares `--execution [e2b|gradio|local]`, so the harness runs end to end on the reader's own machine, and nothing in the README offers a paid tier, a commercial edition or a license key.

BigCodeBench's only adoption signal is GitHub stars across its two repositories, the benchmark harness and its sibling evaluation harness, since neither publishes a PyPI or install figure. Stars alone cannot place a benchmark past a modest reading, because a star is not a use, and the harness repository is now archived and read-only, so its star count is fixed going forward.

• https://api.github.com/repos/bigcode-project/bigcode-evaluation-harness recorded 2026-08-13

  stargazers_count = 1,054, forks 261 for bigcode-project/bigcode-evaluation-harness, the second declared repo

• https://api.github.com/repos/bigcode-project/bigcodebench recorded 2026-08-13

  stargazers_count = 519, forks 75, archived = true, pushed_at 2026-01-03 for bigcode-project/bigcodebench

• https://github.com/bigcode-project/bigcodebench recorded 2026-06-04

  503 stars; leaderboard with 163 models

• https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard recorded 2026-06-04

  active leaderboard ranking evaluated models

BigCodeBench scores code generation by executing more than a thousand library-heavy programming tasks, split into Complete and Instruct variants. Because it covers code generation only, it sits a step below OpenCompass and further behind lm-evaluation-harness, both of which span many kinds of task.

• https://github.com/bigcode-project/bigcodebench recorded 2026-06-04

  1140 tasks, Complete/Instruct splits, execution-based eval

• https://pypi.org/pypi/bigcodebench/json recorded 2026-09-29

  The packaged README says "a new benchmark for code generation with 1140 software-engineering-oriented programming tasks", describes two splits, Complete and Instruct, and shows an evaluation command with "--split [complete|instruct]" and "--subset [full|hard]".

• https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md recorded 2026-08-13

  'a new benchmark for code generation with 1140 software-engineering-oriented programming tasks'; Complete and Instruct splits described; generation and execution-based grading run through `--split [complete|instruct]`.