simple-evals

license

MIT(OSI)

source

public(reference eval implementations)

core features withheld

no

The code is MIT-licensed and public, and MIT is an OSI license. The repository is deprecated and not actively maintained, but the README describes a plain reference library with no paid tier, managed service or license key beside it.

• https://api.github.com/repos/openai/simple-evals/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for openai/simple-evals and returns the MIT text as the LICENSE body.

• https://github.com/openai/simple-evals recorded 2026-06-04

  MIT license; lightweight eval library; deprecation notice (July 2025); ~4.5k stars

• https://raw.githubusercontent.com/openai/simple-evals/main/LICENSE recorded 2026-09-29

  The repository LICENSE file is the MIT License: "MIT License" followed by "Permission is hereby granted, free of charge, to any person obtaining a copy" of the software, with the copyright held by OpenAI.

• https://raw.githubusercontent.com/openai/simple-evals/main/README.md recorded 2026-09-29

  The README opens with a deprecation notice, "will no longer be updated for new models or benchmark results", and calls the library open source: "We are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models". Run instructions use the repo directly ("python -m simple-evals.simple_evals --model <model_name> --examples <num_examples>") and setup needs only pip packages such as "pip install openai". Searched the README for a paid tier, managed service, hosted edition and license key and found none.

simple-evals is run from source with no pip distribution, so there is no download signal, leaving GitHub stars as the only measure available, and a star is not a use. The project is also deprecated and not actively maintained.

• https://api.github.com/repos/openai/simple-evals recorded 2026-08-13

  stargazers_count = 4,598, forks 503, pushed_at 2026-04-22

• https://github.com/openai/simple-evals recorded 2026-06-04

  ~4.5k stars, 490 forks; deprecation notice

• https://raw.githubusercontent.com/openai/simple-evals/main/README.md recorded 2026-08-13

  The README opens with a deprecation notice: "will no longer be updated for new models or benchmark results", and "The repo will continue to host reference implementations for" HealthBench, BrowseComp and SimpleQA.

simple-evals is a lightweight reference library: the README lists nine bundled benchmarks and sampler interfaces for only the OpenAI and Claude APIs. It says the repository is not actively maintained and is not accepting new evals.

• https://github.com/openai/simple-evals recorded 2026-06-04

  bundled benchmark list (MMLU/MATH/GPQA/HumanEval/HealthBench/etc.); zero-shot CoT approach

• https://raw.githubusercontent.com/openai/simple-evals/main/README.md recorded 2026-09-29

  The README lists the bundled evals and their upstream licenses (MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp, HealthBench), says the library emphasizes "the *zero-shot, chain-of-thought* setting", that "We will not be actively maintaining this repository and monitoring PRs and Issues", and that "we're not accepting new evals". Samplers cover only the OpenAI and Claude APIs: "We have implemented sampling interfaces for the following language model APIs".