Model components / Evaluation code

simple-evals

OpenAI's lightweight reference harness used to produce the model cards for GPT-4o, o1, and successors. Includes MMLU, MATH, GPQA, MGSM, DROP, HumanEval and SimpleQA. Picked when researchers want the exact numbers OpenAI reports rather than community variants. 4.5k stars, used as the authoritative reproduction script for GPT scorecards.

openai/simple-evals GitHub repo (MIT, ~4.5k stars). Lightweight zero-shot/CoT eval library bundling MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp, HealthBench. DEPRECATED July 2025; no longer updated for new models/benchmarks; only HealthBench/BrowseComp/SimpleQA reference impls maintained. Confirmed live (deprecated) June 2026.

Openness

5 high confidence

5.0

license: MIT(OSI)
source: public(reference eval implementations)
core-gated: ungated

Fully OSI (MIT); purely a reference-implementation library, no gated core.

https://github.com/openai/simple-evals recorded 2026-06-04
MIT license; lightweight eval library; deprecation notice (July 2025); ~4.5k stars

Adoption

2 low confidence

2.0

Run from source (no pip distribution), so no download signal; ~4.5k stars + 490 forks is the only measure, capped at <=3 by the stars rule and lowered to 2 because the project is deprecated (frozen July 2025) and used mainly as a reference for a few specific benchmarks (HealthBench/BrowseComp/SimpleQA) rather than as a live harness.

https://github.com/openai/simple-evals recorded 2026-06-04
~4.5k stars, 490 forks; deprecation notice

Capability

2 medium confidence

2.0

Deliberately minimal reference harness, now frozen; narrow coverage and feature set vs category leaders (HELM, Inspect AI, lm-eval-harness).

https://github.com/openai/simple-evals recorded 2026-06-04
bundled benchmark list (MMLU/MATH/GPQA/HumanEval/HealthBench/etc.); zero-shot CoT approach

Unchanged since 2026-07-30 (last edited, not re-checked)