HELM

license

Apache-2.0(OSI)

source

public(framework + standardized datasets + web UI + leaderboard code)

governance

Stanford CRFM(academic)

core-gated

ungated

Fully OSI (Apache-2.0); framework, standardized data formats, web UI, and leaderboards all public. HELM is in maintenance mode, which slows the code without withdrawing or gating any of it.

• https://github.com/stanford-crfm/helm recorded 2026-06-04

  Apache-2.0 license; holistic eval framework with standardized datasets, web UI, leaderboards; v0.5.16; 2.8k stars

• https://api.github.com/repos/stanford-crfm/helm/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id Apache-2.0 for stanford-crfm/helm and returns the Apache-2.0 text as the LICENSE body.

• https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md recorded 2026-08-13

  README describes HELM as 'an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford', ships the web leaderboard UI in the same repo, and documents reproducing the published leaderboards from the framework. It carries a maintenance-mode note dated 2026-06-01 and no pricing, paid tier, enterprise edition or license key.

Influence and standard-setting, not install volume, is the signal here: HELM is a widely cited standardized public leaderboard from Stanford CRFM spanning multiple modalities - capabilities, safety, vision, medical - and referenced across the research community. Its PyPI package, crfm-helm, sees only 5,097 downloads a month (about 3,142 at an earlier reading), a volume that would band far lower on its own; most usage runs through the hosted leaderboard and academic citation rather than pip, so downloads understate reach and serve as corroboration rather than as the measure. Level 3 reflects meaningful research and standard-setting adoption short of mass production deployment.

• https://github.com/stanford-crfm/helm recorded 2026-06-04

  multiple public leaderboards (HELM Capabilities/Safety/VHELM/HEIM/MedHELM); Stanford CRFM standardized benchmark

• https://pypistats.org/packages/crfm-helm recorded 2026-06-04

  3,142 PyPI downloads last month (low because usage is leaderboard/citation-driven, not pip)

• https://pypistats.org/api/packages/crfm-helm/recent recorded 2026-08-13

  last_month = 5,097 downloads of the crfm-helm package (last_week 2,327, last_day 714)

• https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md recorded 2026-08-13

  README lists the maintained public leaderboards - HELM Capabilities, HELM Safety, VHELM, HEIM, ToRR, MedHELM and audio - each with its published paper, which is the standard-setting evidence the level rests on.

Frontier-defining on coverage and standardization: the holistic multi-modality leaderboard standard, running the same leaderboard family across text, vision-language, text-to-image, table reasoning, medical and audio, and one of the one or two category leaders. It is weaker than Inspect AI on agentic-eval support but broader on benchmark and modality coverage, which is why the two sit level with each other rather than one above the other. HELM is in maintenance mode, which caps future breadth without narrowing what is already there.

• https://github.com/stanford-crfm/helm recorded 2026-06-04

  standardized datasets, unified model access, metrics beyond accuracy, web UI, multi-modal leaderboards (MMLU-Pro/GPQA/IFEval/WildBench)

• https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md recorded 2026-08-13

  standardized scenario/adapter/metric framework with a web leaderboard UI; leaderboards span text (Capabilities, Safety, ToRR), vision-language (VHELM), text-to-image (HEIM), medical (MedHELM), audio and enterprise benchmarks; maintenance-mode notice dated 2026-06-01.