simple-evals

license

MIT(OSI)

source

public(reference eval implementations)

core-gated

ungated

Fully MIT-licensed, and MIT is an OSI license; purely a reference-implementation library with no gated core. The repo is frozen and its README carries only a deprecation notice, but a frozen repo is still a published one, so the source is still public and the core still ungated.

• https://github.com/openai/simple-evals recorded 2026-06-04

  MIT license; lightweight eval library; deprecation notice (July 2025); ~4.5k stars

• https://api.github.com/repos/openai/simple-evals/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for openai/simple-evals and returns the MIT text as the LICENSE body.

• https://raw.githubusercontent.com/openai/simple-evals/main/README.md recorded 2026-08-13

  README opens with a deprecation notice - 'July 2025: simple-evals will no longer be updated for new models or benchmark results. The repo will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA' - and the eval scripts are in the repo. No paid tier, managed service or license key is offered anywhere in it.

Run from source with no pip distribution, so there is no download signal; the one declared repo carries 4,598 stars and 503 forks, and that is the only measure available. A star count caps a level at 3, and 4,598 bands at 1K-10K stars, level 2 - stars have their own scale because a star is not a download. It sits at 2 rather than higher because the project is deprecated and frozen, and is used mainly as a reference for a few specific benchmarks (HealthBench, BrowseComp, SimpleQA) rather than as a live harness.

• https://github.com/openai/simple-evals recorded 2026-06-04

  ~4.5k stars, 490 forks; deprecation notice

• https://api.github.com/repos/openai/simple-evals recorded 2026-08-13

  stargazers_count = 4,598, forks 503, pushed_at 2026-04-22

• https://raw.githubusercontent.com/openai/simple-evals/main/README.md recorded 2026-08-13

  deprecation notice still first in the README; only HealthBench, BrowseComp and SimpleQA reference implementations remain maintained

A deliberately minimal reference harness, now frozen: the README lists nine bundled benchmarks and says the repo is not actively maintained and takes no new evals. Narrow coverage and feature set next to the category leaders - HELM, Inspect AI and lm-evaluation-harness each sit three rungs above it.

• https://github.com/openai/simple-evals recorded 2026-06-04

  bundled benchmark list (MMLU/MATH/GPQA/HumanEval/HealthBench/etc.); zero-shot CoT approach

• https://raw.githubusercontent.com/openai/simple-evals/main/README.md recorded 2026-08-13

  bundled benchmarks MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp and HealthBench, each with its upstream license; 'We will not be actively maintaining this repository ... we're not accepting new evals'.