lighteval

license

MIT(OSI)

source

public(github.com/huggingface/lighteval)

core features withheld

no

Fully OSI-licensed (MIT), full source public, no managed/commercial tier gating the eval engine.

• https://api.github.com/repos/huggingface/lighteval/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for huggingface/lighteval and returns the MIT text as the LICENSE body.

• https://github.com/huggingface/lighteval recorded 2026-06-04

  MIT license, public source, v0.13.0 (Nov 2025), 1000+ tasks, multi-backend support

• https://pypi.org/pypi/lighteval/json recorded 2026-09-29

  The PyPI-packaged README installs the whole toolkit with pip install lighteval and runs it on local hardware: "Evaluate models on CPU or one or more GPUs using" Accelerate, plus vLLM, SGLang and Nanotron entry points; searched the body for enterprise, license key, pricing, subscription and paid: none, and the endpoint backends are third-party inference providers.

• https://raw.githubusercontent.com/huggingface/lighteval/main/README.md recorded 2026-08-13

  README carries the MIT badge and documents running the whole toolkit locally through `lighteval eval`, `lighteval vllm`, `lighteval sglang` and `lighteval endpoint`, with the 1000+ tasks shipping in the package. The optional endpoints are third-party inference providers, not a paid edition of lighteval. No pricing, enterprise tier or license key appears in it.

lighteval's PyPI downloads are the adoption signal here, and it powers the Hugging Face Open LLM Leaderboard as the reproducible eval suite behind it. That gives it meaningful use within the eval and research community, though a smaller install base than the general-purpose harnesses; GitHub stars corroborate the reading but are not its basis.

• https://github.com/huggingface/lighteval recorded 2026-06-04

  stated as the toolkit powering the HF Open LLM Leaderboard; 2.4k stars (corroboration)

• https://pypistats.org/api/packages/lighteval/recent recorded 2026-08-13

  last_month = 24,962 downloads of the lighteval package (last_week 6,698, last_day 978)

• https://pypistats.org/packages/lighteval recorded 2026-06-04

  ~37,239 lighteval downloads in the last month

• https://raw.githubusercontent.com/huggingface/lighteval/main/README.md recorded 2026-08-13

  README still bills the project as coming 'from Hugging Face's Leaderboard and Evals Team.'

lighteval ships more than a thousand tasks, runs on many model backends including vLLM, SGLang and hosted endpoints, supports custom tasks and metrics, and reaches agentic evaluation through its inspect-ai backend. It sits a step below lm-evaluation-harness, which has been used to report results in hundreds of papers and is the more established reference.

• https://github.com/huggingface/lighteval recorded 2026-06-04

  1000+ tasks, 8+ model backends, reproduces Open LLM Leaderboard evals

• https://huggingface.co/docs/lighteval/index recorded 2026-09-29

  The Hugging Face docs home describes Lighteval as "your all-in-one toolkit for evaluating Large Language Models" with backends including inspect-ai, vllm, sglang, litellm and Text Generation Inference, plus "1000s pre-built evaluation tasks" and custom tasks and metrics.

• https://pypi.org/pypi/lighteval/json recorded 2026-09-29

  The PyPI metadata README says "Lighteval supports **1000+ evaluation tasks** across multiple domains and" languages, listing MMLU, GPQA, GSM8K, MATH, LiveCodeBench, IFEval and MT-Bench.

• https://raw.githubusercontent.com/huggingface/lighteval/main/README.md recorded 2026-08-13

  'Lighteval supports 1000+ evaluation tasks across multiple domains'; backends invoked as `lighteval eval` (inspect-ai, preferred), `vllm`, `sglang` and `endpoint` including HF inference providers; custom task and custom metric authoring documented.