Back to Gap Map Model components / Evaluation code

OLMES

open source / Overall score: 2.0

Ai2's evaluation system behind the OLMo and Tülu releases, built on lm-evaluation-harness with deeper task configuration, instance-level prediction logging and external results storage for faithful reproduction of published numbers. It packages the OLMES standard for evaluating base and instruction-tuned models.

No PyPI package (olmes and oe-eval both 404); install is from source. Last push 2026-03-24, quiet for about five months but not archived, well inside the maintenance window; worth re-checking at the next refresh.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public(GitHub)
core features withheld: no

Apache-2.0 and fully public: the LICENSE body is the stock Apache-2.0 text with no appended terms, and the README documents source-only install (`pip install -e .`) and local evaluation of HF models, with no pricing, paid tier, enterprise edition or license key anywhere in it.

https://api.github.com/repos/allenai/olmes/license recorded 2026-10-06
GitHub's license endpoint reports spdx_id Apache-2.0 and returns the stock Apache-2.0 text with no appended terms
https://raw.githubusercontent.com/allenai/olmes/main/README.md recorded 2026-10-06
README installs from a clone (uv sync or pip install -e .) and runs `olmes` locally against HF models; task configs and the OLMES standard ship in the repo; no pricing, paid tier or license key

Adoption

1 medium confidence

1.0

OLMES has no download channel: neither the olmes nor oe-eval PyPI names resolve, and installation is from source, so GitHub stars are the only adoption signal available. Usage beyond Ai2's own model releases appears limited, and being the eval standard behind OLMo and Tülu speaks to its capability and openness rather than its adoption.

https://api.github.com/repos/allenai/olmes recorded 2026-09-01
stargazers_count = 394, forks 105, pushed_at 2026-03-24, not archived

Capability

3 medium confidence

3.0

Reproducibility-hardened derivative of lm-evaluation-harness with unusually deep task configuration and instance-level logging, but a far narrower task and backend surface than the broad platforms — dozens of curated suites against lighteval's 1000+ tasks and wider backend spread — so one below lighteval, beside bigcodebench.

https://raw.githubusercontent.com/allenai/olmes/main/README.md recorded 2026-10-07
reproduces OLMo/OLMo 2/Tülu 3/OLMES-paper results; deep task configurations, instance-level prediction data, custom metrics, external storage integration; built on lm-evaluation-harness

Verified 2026-09-27