HELM

license

Apache-2.0(OSI)

source

public(framework + standardized datasets + web UI + leaderboard code)

core features withheld

no

Fully OSI (Apache-2.0); framework, standardized data formats, web UI, and leaderboards all public. HELM is in maintenance mode, which slows the code without withdrawing or gating any of it.

• https://api.github.com/repos/stanford-crfm/helm/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id Apache-2.0 for stanford-crfm/helm and returns the Apache-2.0 text as the LICENSE body.

• https://crfm-helm.readthedocs.io/en/latest/maintenance_mode/ recorded 2026-09-29

  Maintenance policy page says HELM entered maintenance mode on June 1, 2026, and that "The HELM code and leaderboards will continue to be available as open-source resources for the community." while "no new features will be added to HELM".

• https://crfm-helm.readthedocs.io/en/latest/reproducing_leaderboards/ recorded 2026-09-29

  Reproducing Leaderboards page says "You can use the HELM package to rerun evaluation runs and recreate a specific public leaderboard." and points to the run-entry and schema files in the public repository at "https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/presentation" and "https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/static"; searched the page for API key, paid and license key: none.

• https://github.com/stanford-crfm/helm recorded 2026-06-04

  Apache-2.0 license; holistic eval framework with standardized datasets, web UI, leaderboards; v0.5.16; 2.8k stars

• https://pypi.org/pypi/crfm-helm/json recorded 2026-09-29

  PyPI metadata for crfm-helm 0.5.16 gives license "Apache License 2.0" with the classifier "License :: OSI Approved :: Apache Software License", and the description calls HELM "an open source Python framework created by the" Stanford Center for Research on Foundation Models, listing a "Web UI for inspecting individual prompts and responses". Searched the metadata for pricing, paid tier, enterprise edition, license key and non-commercial: none.

• https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md recorded 2026-08-13

  README describes HELM as 'an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford', ships the web leaderboard UI in the same repo, and documents reproducing the published leaderboards from the framework. It carries a maintenance-mode note dated 2026-06-01 and no pricing, paid tier, enterprise edition or license key.

HELM's reach shows up in research citation and use of its hosted leaderboards, not in package installs: the crfm-helm PyPI package sees only a fraction of that use because most usage runs through the website and citation instead of pip. Those downloads corroborate the reading rather than measure it directly.

• https://github.com/stanford-crfm/helm recorded 2026-06-04

  multiple public leaderboards (HELM Capabilities/Safety/VHELM/HEIM/MedHELM); Stanford CRFM standardized benchmark

• https://pypistats.org/api/packages/crfm-helm/recent recorded 2026-10-08

  last_month = 1,206 downloads of the crfm-helm package (last_week 284, last_day 45)

• https://pypistats.org/packages/crfm-helm recorded 2026-06-04

  3,142 PyPI downloads last month (low because usage is leaderboard/citation-driven, not pip)

• https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md recorded 2026-10-08

  README lists the maintained public leaderboards: HELM Capabilities, HELM Safety, VHELM, HEIM, ToRR, MedHELM and audio, each with its published paper.

HELM runs the same holistic leaderboard family across text, vision-language, text-to-image, table reasoning, medical and audio, one of the broadest standardized benchmark sets in the category. It has less agentic-eval support than Inspect AI, but wider benchmark and modality coverage, which keeps the two level with each other rather than one ahead.

• https://crfm-helm.readthedocs.io/en/latest/ recorded 2026-09-29

  Docs home lists leaderboards HELM Capabilities, HELM Safety and VHELM, and says the framework offers "Models from various providers accessible through a unified interface" and "Metrics for measuring various aspects beyond accuracy".

• https://crfm-helm.readthedocs.io/en/latest/maintenance_mode/ recorded 2026-09-29

  Maintenance policy page says HELM entered maintenance mode on June 1, 2026, and that "The HELM code and leaderboards will continue to be available as open-source resources for the community." while "no new features will be added to HELM".

• https://github.com/stanford-crfm/helm recorded 2026-06-04

  standardized datasets, unified model access, metrics beyond accuracy, web UI, multi-modal leaderboards (MMLU-Pro/GPQA/IFEval/WildBench)

• https://pypi.org/pypi/crfm-helm/json recorded 2026-09-29

  The package description lists leaderboards for HELM Capabilities, HELM Safety, VHELM, HEIM, ToRR, MedHELM and audio models, and says the framework offers "Models from various providers accessible through a unified interface".

• https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md recorded 2026-08-13

  standardized scenario/adapter/metric framework with a web leaderboard UI; leaderboards span text (Capabilities, Safety, ToRR), vision-language (VHELM), text-to-image (HEIM), medical (MedHELM), audio and enterprise benchmarks; maintenance-mode notice dated 2026-06-01.