lighteval

license

MIT(OSI)

source

public(github.com/huggingface/lighteval)

core-gated

ungated

Fully OSI-licensed (MIT), full source public, no managed/commercial tier gating the eval engine.

• https://github.com/huggingface/lighteval recorded 2026-06-04

  MIT license, public source, v0.13.0 (Nov 2025), 1000+ tasks, multi-backend support

• https://api.github.com/repos/huggingface/lighteval/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for huggingface/lighteval and returns the MIT text as the LICENSE body.

• https://raw.githubusercontent.com/huggingface/lighteval/main/README.md recorded 2026-08-13

  README carries the MIT badge and documents running the whole toolkit locally through `lighteval eval`, `lighteval vllm`, `lighteval sglang` and `lighteval endpoint`, with the 1000+ tasks shipping in the package. The optional endpoints are third-party inference providers, not a paid edition of lighteval. No pricing, enterprise tier or license key appears in it.

lighteval sees 24,962 PyPI downloads in the trailing 30 days, inside the 10K-100K band. It powers the Hugging Face Open LLM Leaderboard and is the reproducible eval suite behind it - meaningful within the eval and research community, but a smaller install base than the general-purpose harnesses. Download volume is what the level rests on; the 2.5k GitHub stars only corroborate.

• https://pypistats.org/packages/lighteval recorded 2026-06-04

  ~37,239 lighteval downloads in the last month

• https://github.com/huggingface/lighteval recorded 2026-06-04

  stated as the toolkit powering the HF Open LLM Leaderboard; 2.4k stars (corroboration)

• https://pypistats.org/api/packages/lighteval/recent recorded 2026-08-13

  last_month = 24,962 downloads of the lighteval package (last_week 6,698, last_day 978)

• https://raw.githubusercontent.com/huggingface/lighteval/main/README.md recorded 2026-08-13

  README still bills the project as coming 'from Hugging Face's Leaderboard and Evals Team', which is the leaderboard tie the corroborating half of this level rests on.

Strong on every dimension that counts here - task coverage, model-backend breadth, reproducibility, and some agentic support by way of the inspect-ai backend. Not a 5, because lm-evaluation-harness remains the broader de-facto standard and is cited in more frontier release reports; lighteval is the Hugging Face-leaderboard-tied counterpart.

• https://github.com/huggingface/lighteval recorded 2026-06-04

  1000+ tasks, 8+ model backends, reproduces Open LLM Leaderboard evals

• https://raw.githubusercontent.com/huggingface/lighteval/main/README.md recorded 2026-08-13

  'Lighteval supports 1000+ evaluation tasks across multiple domains'; backends invoked as `lighteval eval` (inspect-ai, preferred), `vllm`, `sglang` and `endpoint` including HF inference providers; custom task and custom metric authoring documented.