OpenAI Evals API & Dashboard
OpenAIManaged evaluation product inside the OpenAI platform: an Evals API and a dashboard that let developers define eval datasets, run them against OpenAI models, grade with built-in or custom graders, and inspect results across versions.
Being sunset: read-only from 31 October 2026 and shut down on 30 November 2026, with OpenAI directing users to Datasets. Distinct from the openai/evals repository, which is a separate record.
Openness
1 high confidence- license
- Proprietary(SaaS, OpenAI-platform-only)
- source
- closed
The Evals API and dashboard are a closed hosted service that runs only on the OpenAI platform, with no published source. The MIT-licensed openai/evals repository is a separate product on the map.
- https://developers.openai.com/api/docs/deprecations recorded 2026-09-29
The deprecations page states "Oct 31, 2026" existing evals become read-only and "The Evals dashboard and API are scheduled to shut down" on Nov 30, 2026.
- https://developers.openai.com/api/docs/guides/evals recorded 2026-06-04
hosted proprietary Evals API + dashboard on OpenAI platform; sunset dates (read-only Oct 31 2026, shutdown Nov 30 2026)
- https://developers.openai.com/api/docs/guides/evals recorded 2026-09-29
The guide focuses on "configuring evals programmatically using the" Evals API (linked to the OpenAI API reference) or the OpenAI dashboard, and states "OpenAI is deprecating the Evals platform" and "Evals will become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026". No repository, self-hostable build or license for the Evals platform appears on the page; its only self-host link is a nav entry for Agents sandboxes.
Adoption
not assessedOpenAI publishes no usage figures for the Evals API or dashboard specifically, and with the product being sunset there is no other signal that could take the place of one.
- https://developers.openai.com/api/docs/guides/evals recorded 2026-06-04
no usage/customer figures; product on sunset path
- https://developers.openai.com/api/docs/guides/evals recorded 2026-09-29
Searched the body for a customer count, developer count or eval run count: none. The only status line is "OpenAI is deprecating the Evals platform", with no usage figure.
Capability
2 medium confidenceThe Evals API gives developers already on OpenAI a managed way to grade prompts against their own datasets. It works only with OpenAI models, ships no benchmark library, and is being deprecated.
- https://developers.openai.com/api/docs/guides/evals recorded 2026-06-04
grading methods, dataset upload, run execution vs OpenAI APIs, dashboard + cost analysis
- https://developers.openai.com/api/docs/guides/evals recorded 2026-09-29
The guide says the graders "determine if the model output is correct" and that evals can be configured "in the OpenAI dashboard"; runs take a Chat Completions messages array or a Responses input.
- https://developers.openai.com/api/docs/guides/graders recorded 2026-09-29
The graders guide says "Graders are specified in JSON format, and there are several types" including String check, Text similarity, Score model grader and Python code execution.
Verified 2026-09-29