Inspect AI

license

MIT(OSI)

source

public(full framework, 200+ evals)

governance

public-sector(UK AISI)+Meridian Labs

core-gated

ungated

Fully MIT-licensed and government-backed, with no proprietary core and the whole pipeline open.

• https://github.com/UKGovernmentBEIS/inspect_ai recorded 2026-06-04

  MIT license; framework for LLM evals with 200+ pre-built evals; UK AISI maintainer; 2.2k stars, 5,886 commits

• https://api.github.com/repos/UKGovernmentBEIS/inspect_ai/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for UKGovernmentBEIS/inspect_ai and returns the MIT text as the LICENSE body.

• https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_ai/main/README.md recorded 2026-08-13

  README is a build-from-source guide - clone the repo, `pip install -e ".[dev]"`, build the React web UI from the checked-in submodule, render the docs locally. It names UK AISI as the creator and points at 'over 200 pre-built evaluations ready to run on any model'. There is no pricing, no hosted tier, no enterprise edition and no license key anywhere in it.

Adoption by frontier labs is the strongest signal a harness can carry, and UK AISI open-sourced Inspect as its evaluations platform and reports that it 'is used by other governments and some of the leading AI developers'. The AISI post does not name the individual labs; what it does report is evaluations on 16 models in AISI's first year, so the basis is what that post asserts rather than any specific roster. PyPI volume corroborates - 9,278,867 inspect-ai downloads in the trailing 30 days on pypistats, against ~9.86M on pepy - but that traffic is inflated by CI and dependency installs, which is why lab adoption rather than download count is the basis. Level 4 rather than 5, because the human-operator base of eval engineers at labs and at AISI is far smaller than mass-market scale.

• https://www.aisi.gov.uk/blog/our-first-year recorded 2026-08-13

  'we open sourced our evaluations platform, Inspect ... now it is used by other governments and some of the leading AI developers'. The post reports evaluations on 16 models in AISI's first year, and carries no 'thousands of evaluations' figure.

• https://inspect.aisi.org.uk/ recorded 2026-06-04

  framework for frontier AI evaluations by UK AISI + Meridian Labs; adopted by major labs

• https://pepy.tech/projects/inspect-ai recorded 2026-06-04

  87.89M total / 9,856,617 last-30-day PyPI downloads (corroboration)

• https://pypistats.org/api/packages/inspect-ai/recent recorded 2026-08-13

  last_month = 9,278,867 downloads of the inspect-ai package (last_week 1,494,033, last_day 262,149)

Frontier-defining for this category: the broadest backend coverage, first-class agentic-eval support, and de-facto-standard status among frontier-safety labs put it among the one or two leaders. The providers page is wider still, also listing Mistral, DeepSeek, Moonshot, Perplexity, SageMaker, Fireworks, SambaNova, SGLang and TransformerLens behind the same interface.

• https://inspect.aisi.org.uk/models.html recorded 2026-06-04

  single interface over OpenAI/Anthropic/Google/xAI/Bedrock/Azure/local vLLM/Ollama/llama-cpp

• https://github.com/UKGovernmentBEIS/inspect_ai recorded 2026-06-04

  200+ pre-built evals; tool use, multi-turn, model-graded eval components

• https://inspect.aisi.org.uk/models.html recorded 2026-08-13

  One naming convention (`openai/gpt-4o`, `anthropic/claude-sonnet-4-0`) over lab APIs (OpenAI, Anthropic, Google, Grok, Mistral, DeepSeek, Moonshot, Perplexity), cloud APIs (Bedrock, SageMaker, Azure AI), hosted open models (Groq, Together, Fireworks, Cloudflare, HF Inference Providers, SambaNova) and local ones (HF, vLLM, Ollama, llama-cpp-python, SGLang, TransformerLens).

• https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_ai/main/README.md recorded 2026-08-13

  README: built-in components for prompt engineering, tool usage, multi-turn dialog and model-graded evaluations, plus 'a collection of over 200 pre-built evaluations ready to run on any model'.