Vals AI

license

proprietary(the Vals Index platform and its held-out sets)+AGPL-3.0(OSI, vals-ai/Valkyrie)+MIT(OSI, vals-ai/model-library)

source

partial(harness repos public, Index runtime and datasets do not ship)

benchmarks

proprietary/private-held-out(Vals Index on non-public datasets)

eval-harness

published(Valkyrie AGPL-3.0 + model-library MIT)

A third-party eval platform whose benchmarks are deliberately held private to prevent dataset leakage, with part of the infrastructure now published. The about page states 'We have built, and even open-sourced, the infrastructure we rely on to run these evaluations reproducibly and at scale across labs. This includes Valkyrie, a distributed system to run agentic benchmarks, and our model library, a free standard API to call models with'. Both repos are live, public and actively pushed - vals-ai/Valkyrie under AGPL-3.0 and vals-ai/model-library under MIT - so neither the source nor the eval harness is entirely closed. What is closed is the part the tier turns on: scores are still 'based on our privately held test sets', the Vals Index runtime is still a hosted service with no self-hostable implementation, and the held-out datasets do not ship. A repo that exists while the runtime does not is only a partial source, which is source-available rather than closed.

• https://www.vals.ai recorded 2026-06-04

  proprietary SaaS eval platform, 'proprietary benchmarks' require contact, no GitHub/self-host option

• https://www.vals.ai/about recorded 2026-06-04

  runs evaluations in-house on private/held-out datasets as neutral third party

• https://www.vals.ai/about recorded 2026-08-13

  Vals describes itself as 'the independent evaluator of artificial intelligence' running evaluations in-house on 'privately held test sets'. It ALSO states it has open-sourced Valkyrie, a distributed system for running agentic benchmarks, and a model library offering a free standard API for calling models, both linked to GitHub.

• https://www.vals.ai/about recorded 2026-08-14

  'We have built, and even open-sourced, the infrastructure we rely on to run these evaluations reproducibly and at scale across labs. This includes Valkyrie, a distributed system to run agentic benchmarks, and our model library, a free standard API to call models with.' Beside it, 'Scores are based on our privately held test sets to preserve the integrity and signal of our results' - the harness ships, the sets and the Index runtime do not.

• https://api.github.com/repos/vals-ai/valkyrie recorded 2026-08-14

  Repo metadata for vals-ai/Valkyrie - license spdx_id AGPL-3.0, public, not archived, 31 stars, pushed 2026-08-14, described as 'Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark'. A live published harness rather than an abandoned mirror.

• https://api.github.com/repos/vals-ai/model-library recorded 2026-08-14

  Repo metadata for vals-ai/model-library - license spdx_id MIT, public, not archived, 21 stars, pushed 2026-07-30, 'Simple provider agnostic LLM gateway'.