OLMES

license

Apache-2.0(OSI)

source

public(GitHub)

core-gated

ungated

Apache-2.0 and fully public: the LICENSE body is the stock Apache-2.0 text with no appended terms, and the README documents source-only install (`pip install -e .`) and local evaluation of HF models, with no pricing, paid tier, enterprise edition or license key anywhere in it.

• https://api.github.com/repos/allenai/olmes/license recorded 2026-09-01

  GitHub's license endpoint reports spdx_id Apache-2.0 and returns the stock Apache-2.0 text with no appended terms

• https://raw.githubusercontent.com/allenai/olmes/main/README.md recorded 2026-09-01

  README installs from a clone (uv sync or pip install -e .) and runs `olmes` locally against HF models; task configs and the OLMES standard ship in the repo; no pricing, paid tier or license key

394 GitHub stars, which bands at level 1 on the stars scale (<1K). No download channel exists — no PyPI package (olmes and oe-eval both 404 this session), install is from source — and external usage beyond Ai2's own model releases is limited, so stars are the honest instrument. The provenance (the eval standard behind OLMo and Tülu) is capability and openness signal, not adoption.

• https://api.github.com/repos/allenai/olmes recorded 2026-09-01

  stargazers_count = 394, forks 105, pushed_at 2026-03-24, not archived

Reproducibility-hardened derivative of lm-evaluation-harness with unusually deep task configuration and instance-level logging, but a far narrower task and backend surface than the broad platforms — dozens of curated suites against lighteval's 1000+ tasks and wider backend spread — so one below lighteval, beside bigcodebench.

• https://raw.githubusercontent.com/allenai/olmes/main/README.md recorded 2026-09-01

  reproduces OLMo/OLMo 2/Tülu 3/OLMES-paper results; deep task configurations, instance-level prediction data, custom metrics, external storage integration; built on lm-evaluation-harness