Model components / Evaluation code

OpenAI Evals

Framework for evaluating language models and the systems built on them, published with a registry of community-contributed benchmarks as the 2023 reference implementation for eval-driven development. It supports custom eval authoring and model-graded grading, and runs against private data as well as the bundled registry.

Development has effectively stopped, with no push since April 2026, as work migrated to OpenAI's hosted Evals API, itself now on a sunset path. Bundled eval datasets are carved out of the repository terms. Verified 2026-08-13 via the openai/evals README, its LICENSE.md and the repository metadata.

Openness

5 high confidence

5.0

license: MIT(OSI)
source: public(full framework + registry)
per-dataset-licenses: mixed(CC/CC0/Apache for bundled data)
core-gated: ungated

Framework code is fully OSI (MIT); the included eval datasets carry their own open licenses, which is standard for a harness. GitHub's own detector reports NOASSERTION because of the appended per-dataset section, which is a detector artifact rather than a second license over the framework code.

https://github.com/openai/evals/blob/main/LICENSE.md recorded 2026-06-04
MIT License text (copyright 2023 OpenAI)
https://github.com/openai/evals recorded 2026-06-04
open framework for evaluating LLMs with open benchmark registry; MIT badge; 18.6k stars
https://raw.githubusercontent.com/openai/evals/main/LICENSE.md recorded 2026-08-13
LICENSE.md opens 'MIT License / Copyright (c) 2023 OpenAI' with the full unmodified MIT body, then 'NOTE: This license applies to all parts of this repository except for the datasets specified below' followed by the per-dataset license table.
https://raw.githubusercontent.com/openai/evals/main/README.md recorded 2026-08-13
README documents cloning the repo, `pip install -e .`, and pulling the bundled eval registry over Git-LFS, so the whole framework and its benchmark registry run from the published source. The only requirement it adds is the reader's own OPENAI_API_KEY. No pricing, paid tier, enterprise edition or license key appears in it.

Adoption

3 low confidence

3.0

The repo is run from source rather than pip-installed, so no download count exists for it. Its 19,161 GitHub stars and 3,052 forks are the only honest signal, and a star count caps a level at 3; 19,161 bands at >10K stars on the stars scale, which is a separate vocabulary from downloads because a star is not a download. Historically influential as one of the first open eval frameworks, it has been largely superseded by the hosted Evals API and by newer harnesses, and the repo has gone months without a commit.

https://github.com/openai/evals recorded 2026-06-04
18.6k stars, 3k forks, 691 commits
https://api.github.com/repos/openai/evals recorded 2026-08-13
stargazers_count = 19,161, forks 3,052, pushed_at 2026-04-14, archived = false

Capability

3 medium confidence

3.0

A solid general-purpose authoring framework - the README describes a registry of evals plus custom-eval authoring - but not frontier on task coverage, backend breadth or agentic-eval support next to category leaders such as lm-evaluation-harness, and its maintenance has slowed.

https://github.com/openai/evals recorded 2026-06-04
framework + registry of benchmarks, custom eval authoring, model-graded grading
https://raw.githubusercontent.com/openai/evals/main/README.md recorded 2026-08-13
README: 'We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals'; private evals over your own data supported. No agentic-eval harness and no multi-backend abstraction described.

Verified 2026-08-13