Model components / Evaluation code

lighteval

Lightweight successor to the eval harness HF used internally; supports vLLM, transformers, TGI and OpenAI-compatible backends with a focus on speed and easy custom metric authoring. Picked when teams want a fast, hackable runner that integrates cleanly with the HF ecosystem (datasets, accelerate). 2.4k stars, 120 contributors, 17 commits in last 90 days; powers parts of the Open LLM Leaderboard v2.

lighteval v0.13.0 (24 Nov 2025), latest GitHub release; confirmed live on GitHub June 2026. All-in-one LLM eval toolkit from HF's Leaderboard & Evals team; powers the HF Open LLM Leaderboard. Columbia extension (model.code.evaluation) caveat applies.

Openness

5 high confidence

5.0

license: MIT(OSI)
source: public(github.com/huggingface/lighteval)
core-gated: ungated

Fully OSI-licensed (MIT), full source public, no managed/commercial tier gating the eval engine.

https://github.com/huggingface/lighteval recorded 2026-06-04
MIT license, public source, v0.13.0 (Nov 2025), 1000+ tasks, multi-backend support

Adoption

2 medium confidence

2.0

~37,239 PyPI downloads in the last month (pypistats). Powers the HF Open LLM Leaderboard and is the reproducible eval suite behind it; meaningful in the eval/research community but a smaller install base than general-purpose harnesses. 2.4k GitHub stars corroborate only. Headline = PyPI volume (primary).

https://pypistats.org/packages/lighteval recorded 2026-06-04
~37,239 lighteval downloads in the last month
https://github.com/huggingface/lighteval recorded 2026-06-04
stated as the toolkit powering the HF Open LLM Leaderboard; 2.4k stars (corroboration)

Capability

4 high confidence

4.0

Strong across the recipe's feature-matrix dimensions (coverage, backend breadth, reproducibility, some agentic support). Not a 5; lm-evaluation-harness remains the broader de-facto standard cited in more frontier release reports; lighteval is the HF-leaderboard-tied counterpart.

https://github.com/huggingface/lighteval recorded 2026-06-04
1000+ tasks, 8+ model backends, reproduces Open LLM Leaderboard evals

Unchanged since 2026-07-30 (last edited, not re-checked)