OpenAI Evals

license

MIT(OSI)

source

public(full framework + registry)

per-dataset-licenses

mixed(CC/CC0/Apache for bundled data)

core features withheld

no

Framework code is fully OSI (MIT); the included eval datasets carry their own open licenses, which is standard for a harness. GitHub's own detector reports NOASSERTION because of the appended per-dataset section, which is a detector artifact rather than a second license over the framework code.

• https://github.com/openai/evals recorded 2026-06-04

  open framework for evaluating LLMs with open benchmark registry; MIT badge; 18.6k stars

• https://github.com/openai/evals/blob/main/LICENSE.md recorded 2026-06-04

  MIT License text (copyright 2023 OpenAI)

• https://pypi.org/pypi/evals/json recorded 2026-09-29

  The packaged README states "you are agreeing to make your evaluation logic and data under the same MIT license as this repository" for contributions.

• https://pypi.org/pypi/evals/json recorded 2026-09-29

  The packaged README says "Our evals registry is stored using" Git-LFS and that installing from the cloned repo uses "pip install -e ." so the framework and registry run from source.

• https://pypi.org/pypi/evals/json recorded 2026-09-29

  The packaged README says "To run evals, you will need to set up and specify your" OpenAI API key and warns of API costs; it mentions no paid edition, enterprise build or license key (searched the body for enterprise, license key, paid tier: none).

• https://raw.githubusercontent.com/openai/evals/main/LICENSE.md recorded 2026-08-13

  LICENSE.md opens 'MIT License / Copyright (c) 2023 OpenAI' with the full unmodified MIT body, then 'NOTE: This license applies to all parts of this repository except for the datasets specified below' followed by the per-dataset license table.

• https://raw.githubusercontent.com/openai/evals/main/README.md recorded 2026-08-13

  README documents cloning the repo, `pip install -e .`, and pulling the bundled eval registry over Git-LFS, so the whole framework and its benchmark registry run from the published source. The only requirement it adds is the reader's own OPENAI_API_KEY. No pricing, paid tier, enterprise edition or license key appears in it.

OpenAI Evals is run from source rather than pip-installed, so no download count exists for it, leaving GitHub stars and forks as the only honest usage signal available, and a star is not a use. The framework was influential as one of the first open eval harnesses, but it has since given way to OpenAI's hosted Evals API and to newer harnesses, and the repository has gone months without a commit.

• https://api.github.com/repos/openai/evals recorded 2026-08-13

  stargazers_count = 19,161, forks 3,052, pushed_at 2026-04-14, archived = false

• https://github.com/openai/evals recorded 2026-06-04

  18.6k stars, 3k forks, 691 commits

OpenAI Evals pairs a registry of evals with custom-eval authoring and model-graded scoring. Its task coverage, backend breadth and agentic support are well behind lm-evaluation-harness, and maintenance has slowed.

• https://github.com/openai/evals recorded 2026-06-04

  framework + registry of benchmarks, custom eval authoring, model-graded grading

• https://pypi.org/pypi/evals/json recorded 2026-09-29

  The packaged README says "We offer an existing registry of evals to test different dimensions of OpenAI models" and supports custom and model-graded evals plus a Completion Function Protocol for "prompt chains or tool-using agents".

• https://raw.githubusercontent.com/openai/evals/main/README.md recorded 2026-08-13

  README: 'We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals'; private evals over your own data supported. No agentic-eval harness and no multi-backend abstraction described.