Vals AI

license

proprietary(the Vals Index and its held-out test sets)+AGPL-3.0(OSI, vals-ai/Valkyrie)+MIT(OSI, vals-ai/model-library)

source

partial(harness repos public, held-out test sets not published)

benchmarks

proprietary/private-held-out(Vals Index on non-public datasets)

eval-harness

published(Valkyrie AGPL-3.0 + model-library MIT)

Vals AI is a proprietary evaluation service, but part of its infrastructure is open: Valkyrie, its agentic-benchmark harness, is AGPL-3.0, and its model library, a provider-agnostic API gateway, is MIT. The test sets behind its scores are held privately and never published.

• https://api.github.com/repos/vals-ai/model-library recorded 2026-08-14

  Repo metadata for vals-ai/model-library - license spdx_id MIT, public, not archived, 21 stars, pushed 2026-07-30, 'Simple provider agnostic LLM gateway'.

• https://api.github.com/repos/vals-ai/valkyrie recorded 2026-08-14

  Repo metadata for vals-ai/Valkyrie - license spdx_id AGPL-3.0, public, not archived, 31 stars, pushed 2026-08-14, described as 'Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark'. A live published harness rather than an abandoned mirror.

• https://raw.githubusercontent.com/vals-ai/model-library/main/LICENSE recorded 2026-09-29

  Raw LICENSE of vals-ai/model-library is the "MIT License" with "Copyright (c) 2025 Vals AI, Inc.", so the model library is MIT.

• https://raw.githubusercontent.com/vals-ai/Valkyrie/main/docs/HOSTED_MODE.md recorded 2026-09-29

  Valkyrie docs describe a self-hosted mode that needs no Vals service: "Run your own infrastructure end-to-end, including your own tracker service deployment." Hosted mode is an option that uses "a Descope API key" from Vals; self-hosting is the alternative.

• https://raw.githubusercontent.com/vals-ai/Valkyrie/main/LICENSE recorded 2026-09-29

  Raw LICENSE of vals-ai/Valkyrie is the full text of "GNU AFFERO GENERAL PUBLIC LICENSE" "Version 3, 19 November 2007", so the harness is AGPL-3.0.

• https://www.vals.ai recorded 2026-06-04

  proprietary SaaS eval platform, 'proprietary benchmarks' require contact, no GitHub/self-host option

• https://www.vals.ai/about recorded 2026-06-04

  runs evaluations in-house on private/held-out datasets as neutral third party

• https://www.vals.ai/about recorded 2026-08-13

  Vals describes itself as 'the independent evaluator of artificial intelligence' running evaluations in-house on 'privately held test sets'. It ALSO states it has open-sourced Valkyrie, a distributed system for running agentic benchmarks, and a model library offering a free standard API for calling models, both linked to GitHub.

• https://www.vals.ai/about recorded 2026-09-29

  The About page says the evaluations run on held-out sets and that the run infrastructure is public: "Scores are based on our privately held test sets to preserve the integrity and signal of our results." and "We have built, and even open-sourced, the infrastructure we rely on to run these evaluations reproducibly and at scale across labs." The harness ships; the test sets do not.

• https://www.vals.ai/benchmarks recorded 2026-09-29

  The benchmarks page tags the Vals Index "Proprietary" and describes its own sets as private: "Private question-answer benchmark over Canadian court-cases." and "A private benchmark evaluating understanding of long-context credit agreements".

Vals AI's benchmarks page tracks models across the Vals Index, a Multimodal Index and vertical suites spanning law, finance, healthcare, math and coding. Vals publishes no user, traffic or customer count, so this reflects its standing as a benchmark provider rather than a measured usage figure.

• https://www.vals.ai/benchmarks recorded 2026-06-04

  live multi-domain leaderboards (Vals Index, Multimodal Index) tracking 24+ frontier models, updated June 2026

• https://www.vals.ai/benchmarks recorded 2026-08-13

  43 models tested; Vals Index, RSI Index and Vals Multimodal Index plus legal, finance, healthcare, math, academic, education, coding, cybersecurity and games benchmarks. No usage, traffic or customer figure is published on the page.

• https://www.vals.ai/benchmarks recorded 2026-09-29

  The leaderboard index lists the Vals Index, Vals RSI Index, Web Search Index and Vals Multimodal Index plus legal, finance, healthcare, math, science, education, voice, coding, games and social mobility suites, e.g. "Evaluating agents on legal research tasks across diverse areas of US law". It gives per-benchmark model counts only; a search for customers and users finds no usage, traffic or customer figure.

Vals AI covers finance, legal, healthcare, math, science, education, coding, security and games suites, plus the Vals Index, RSI Index and Multimodal Index. Many of them run on privately held test sets that outsiders cannot rerun, though the Valkyrie and model-library repositories are published.

• https://www.vals.ai recorded 2026-06-04

  Vals Index + Multimodal Index + domain benchmarks (finance/law/healthcare/coding/math/education) incl TaxEval, CorpFin, MedCode, LegalBench

• https://www.vals.ai recorded 2026-08-13

  Vals Index and Vals Multimodal Index over finance (CorpFin v2, Finance Agent v2, TaxEval v2), legal (LegalBench, Case Law v2, Legal Research Bench), healthcare (MedCode, MedScribe, MedQA), coding (SWE-bench Verified, Terminal-Bench, Vibe Code Bench, IOI, LiveCodeBench), math (ProofBench, AIME, GPQA Diamond) and education (SAGE, MMLU Pro, MMMU). 'We run all of our own evaluations and create many of our benchmarks in-house'.

• https://www.vals.ai recorded 2026-09-29

  The home page states "run all of our own evaluations and create many of our benchmarks" and shows new model releases from several labs each "evaluated across our benchmark suite".

• https://www.vals.ai/about recorded 2026-09-29

  The About page says "Scores are based on our privately held test sets to preserve the integrity and signal of our results." and "This prevents training to the test set, unlike in open-source benchmarks."

• https://www.vals.ai/benchmarks recorded 2026-09-29

  The benchmarks page covers finance ("Evaluating agents on core financial analyst tasks", "A private benchmark evaluating understanding of long-context credit agreements"), healthcare ("Can models support the medical billing process?"), legal ("Evaluating agents on legal research tasks across diverse areas of US law"), math, science, education, coding, security ("Can AI agents find security bugs and patch them?") and games, alongside the composite Vals Index, RSI Index and Multimodal Index, and lists the suites "CorpFin v2", "MedCode", "LegalBench" and "Finance Agent v2".