lm-evaluation-harness

license

MIT(OSI)

source

public(GitHub)

core features withheld

no

MIT-licensed and fully public, and MIT is an OSI license, so this is open source.

• https://api.github.com/repos/EleutherAI/lm-evaluation-harness/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for EleutherAI/lm-evaluation-harness and returns the MIT text as the LICENSE.md body.

• https://github.com/EleutherAI/lm-evaluation-harness recorded 2026-06-04

  MIT license, public source, v0.4.12 (May 2026)

• https://pypi.org/pypi/lm-eval/json recorded 2026-09-29

  PyPI metadata for lm-eval 0.4.13 lists the license as "MIT" and the packaged README tells users to install from source with "git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness" and pip install -e.

• https://pypi.org/pypi/lm-eval/json recorded 2026-09-29

  Searched the metadata and README for pricing, enterprise and license key: none; the only commercial mention is "Support for commercial APIs including" OpenAI, for evaluating third-party hosted models.

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-08-13

  README documents installing from the repo and running `lm-eval` locally against local weights, vLLM, and commercial APIs; the whole task library ships in `lm_eval/tasks`. Nothing in the 53KB of it mentions pricing, a paid tier, an enterprise edition or a license key - the only commercial reference is that the harness can call third-party model APIs.

lm-evaluation-harness's adoption is measured from its PyPI downloads, and standing as a de facto standard corroborates that measure: it backs the Hugging Face Open LLM Leaderboard and appears in hundreds of papers and model release reports as how authors report reproducible benchmark numbers.

• https://github.com/EleutherAI/lm-evaluation-harness recorded 2026-06-04

  backend for HF Open LLM Leaderboard; used in hundreds of papers

• https://pypistats.org/api/packages/lm-eval/recent recorded 2026-08-13

  last_month = 1,658,820 downloads of the lm-eval package (last_week 429,834, last_day 63,998)

• https://pypistats.org/packages/lm-eval recorded 2026-06-04

  ~1,571,581 monthly PyPI downloads

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-08-13

  README records that the Open LLM Leaderboard task group ships in the repo and that the harness backs it; 13.6k stars on the repo corroborate.

More than 60 academic benchmarks with hundreds of subtasks run through one interface over Hugging Face Transformers, vLLM, SGLang, NeMo, Megatron-LM, hosted APIs and local servers. The harness powered the Open LLM Leaderboard and has been used in hundreds of papers to report reproducible benchmark numbers.

• https://github.com/EleutherAI/lm-evaluation-harness recorded 2026-06-04

  60+ benchmarks, multi-backend support, Open LLM Leaderboard backend

• https://pypi.org/pypi/lm-eval/json recorded 2026-09-29

  The packaged README describes the harness as offering "Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented" and says it has been used in "hundreds of papers".

• https://pypi.org/pypi/lm-eval/json recorded 2026-09-29

  The packaged README lists backends with "Support for commercial APIs including" OpenAI and TextSynth, vLLM and SGLang support, and NVIDIA NeMo and Megatron-DeepSpeed integrations.

• https://pypi.org/pypi/lm-eval/json recorded 2026-09-29

  The packaged README says the Open LLM Leaderboard tasks ship in the repo: "Open LLM Leaderboard tasks have been added ! You can find them under the" leaderboard folder.

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-08-13

  'Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented'; model types cover HF Transformers, vLLM, SGLang, NeMo, Megatron-LM, commercial APIs and local inference servers; leaderboard task group present.