lm-evaluation-harness

license

MIT(OSI)

source

public(GitHub)

core-gated

ungated

MIT-licensed and fully public, and MIT is an OSI license, so this is open source.

• https://github.com/EleutherAI/lm-evaluation-harness recorded 2026-06-04

  MIT license, public source, v0.4.12 (May 2026)

• https://api.github.com/repos/EleutherAI/lm-evaluation-harness/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for EleutherAI/lm-evaluation-harness and returns the MIT text as the LICENSE.md body.

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-08-13

  README documents installing from the repo and running `lm-eval` locally against local weights, vLLM, and commercial APIs; the whole task library ships in `lm_eval/tasks`. Nothing in the 53KB of it mentions pricing, a paid tier, an enterprise edition or a license key - the only commercial reference is that the harness can call third-party model APIs.

The lm-eval package draws 1,658,820 PyPI downloads in the trailing 30 days, which bands at level 4 (1M-10M). Downloads are the primary measure here, and standard status corroborates them: the harness backs the Hugging Face Open LLM Leaderboard - the README names the leaderboard task group - and is used in hundreds of papers and model release reports, the strongest 'cited as standard' signal in the category.

• https://pypistats.org/packages/lm-eval recorded 2026-06-04

  ~1,571,581 monthly PyPI downloads

• https://github.com/EleutherAI/lm-evaluation-harness recorded 2026-06-04

  backend for HF Open LLM Leaderboard; used in hundreds of papers

• https://pypistats.org/api/packages/lm-eval/recent recorded 2026-08-13

  last_month = 1,658,820 downloads of the lm-eval package (last_week 429,834, last_day 63,998)

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-08-13

  README records that the Open LLM Leaderboard task group ships in the repo and that the harness backs it; 13.6k stars on the repo corroborate.

The widest task and backend coverage in the category, and the standardization reference for LLM evaluation. The README states 'Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented', and lists the spread of model backends behind them.

• https://github.com/EleutherAI/lm-evaluation-harness recorded 2026-06-04

  60+ benchmarks, multi-backend support, Open LLM Leaderboard backend

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-08-13

  'Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented'; model types cover HF Transformers, vLLM, SGLang, NeMo, Megatron-LM, commercial APIs and local inference servers; leaderboard task group present.