AI Potluck
Model components / Evaluation code

OpenAI Evals

OpenAI

Framework for evaluating LLMs and LLM systems plus a registry of community-contributed benchmarks, published as the original 2023 reference implementation for eval-driven development. Picked because it is the canonical starting point referenced in OpenAI's own docs and many derivative harnesses. 18.5k stars but development has effectively slowed (2 commits in last 90 days) as work migrated to the closed OpenAI Dashboard Evals API.

openai/evals GitHub repo (MIT, 18.6k stars, ~691 commits). Open-source framework for evaluating LLMs with a community benchmark registry. Confirmed live on GitHub June 2026; bundled eval datasets carry their own per-dataset licenses (CC/Apache/CC0).

Openness

5 high confidence
5.0
license
MIT(OSI)
source
public(full framework + registry)
per-dataset-licenses
mixed(CC/CC0/Apache for bundled data)
core-gated
ungated

Framework code is fully OSI (MIT); the included eval datasets carry their own open licenses, which is standard for a harness.

Adoption

3 low confidence
3.0

No PyPI/download count available for this repo (it is run from source, not pip-installed); 18.6k GitHub stars + 3k forks is the only honest signal, so capped at level 3 per the stars-fallback rule. Historically influential as one of the first open eval frameworks but largely superseded by the hosted Evals API and by newer harnesses.

Capability

3 medium confidence
3.0

Solid general-purpose authoring framework but not frontier on task coverage, backend breadth, or agentic-eval support relative to category leaders.

Unchanged since 2026-07-30 (last edited, not re-checked)