Scale Evaluation

platform

closed(proprietary hosted SaaS)

datasets

closed(private, unpublished SEAL benchmarks kept private to prevent gaming/training contamination)

methodology

partially-disclosed(leaderboard transparency) but no open eval code or self-hostable engine

source

closed(proprietary hosted platform, VPC deployment offered but no published source)

Proprietary by design: the deliberately private benchmark datasets and the hosted expert-eval workforce are the moat, and there is no open eval code. The platform page mentions a 'largely open-source technology toolkit' in passing but names no repository and offers nothing downloadable, so nothing about the source is actually open.

• https://scale.com/genai-platform recorded 2026-06-04

  Scale GenAI Platform: automated + expert-human evaluation 'Trust Feedback Loop'; test against proprietary benchmark datasets

• https://scale.com/blog/leaderboard recorded 2026-06-04

  SEAL Leaderboards use curated PRIVATE/unpublished datasets that 'cannot be gamed'

• https://scale.com/genai-platform recorded 2026-08-11

  The platform page offers 'Deploy securely within your own VPC. Full support for AWS, Azure, and GCP' - a customer-cloud deployment of the proprietary product, not published source. No repository, no self-hostable open implementation, and no license to the evaluation code is offered anywhere on the page.

• https://scale.com/genai-platform recorded 2026-08-13

  The platform page offers 'Deploy securely within your own VPC. Full support for AWS, Azure, and GCP' and names SEAL as an in-house frontier benchmarking lab. It refers to a 'largely open-source technology toolkit' but identifies no repository and offers no downloadable source or license to the evaluation engine.

• https://scale.com/blog/leaderboard recorded 2026-08-13

  'Scale's proprietary datasets will remain private and unpublished, ensuring they cannot be exploited', and the leaderboards use 'curated private datasets that can't be gamed'. No eval code is published alongside them.

The SEAL Leaderboards are widely cited and described as 'trusted by leading AI labs' as a third-party evaluator, and SEAL Showdown reports 'millions of real-world conversations' across 100+ countries, 70 languages and 200 professional domains. Scale discloses no paying-customer count for the Evaluation platform specifically, so this is reported traction of the eval surface rather than a verified user count, and the level is directional.

• https://scale.com/blog/showdown recorded 2026-06-04

  SEAL Showdown built on millions of real-world conversations, 100+ countries, 70 languages, 200 domains

• https://scale.com/blog/leaderboard recorded 2026-06-04

  third-party evaluator trusted by leading AI labs

• https://scale.com/blog/showdown recorded 2026-08-13

  'SEAL Showdown is based on millions of real-world conversations from Scale's diverse global contributor network' spanning 'over 100 countries, 70 languages, and 200 professional domains'.

• https://scale.com/blog/leaderboard recorded 2026-08-13

  Scale is described as 'a third-party model evaluator trusted by leading AI labs'. No user, customer or run count is published.

Very broad professional eval coverage - multi-domain private benchmarks, an expert human workforce, adversarial red-teaming and real-preference data - and strong commercial capability. Not a 5: it is closed, non-reproducible by design, and not a community-standard harness in the open-eval-code sense. One caveat on the evidence: the SEAL leaderboard announcement cited here covers four domains - coding, instruction following, math (GSM1k) and multilinguality - and does not itself establish the reasoning, agentic, multimodal and safety coverage claimed alongside them. That breadth rests instead on the platform page (auto-generated eval datasets, adversarial vulnerability scans, expert human eval) plus the SEAL Showdown real-preference stream, which together are enough to support this score.

• https://scale.com/blog/leaderboard recorded 2026-06-04

  SEAL multi-domain expert-driven leaderboards (coding/IF/math/multilingual/reasoning/agentic/safety)

• https://scale.com/genai-platform recorded 2026-06-04

  auto-generated eval datasets + proprietary benchmarks + adversarial vulnerability scans + expert human eval

• https://scale.com/blog/leaderboard recorded 2026-08-13

  The post names four launch domains - coding, instruction following, math (GSM1k) and multilinguality - with a stated intention to add more. It does NOT list reasoning, agentic or safety leaderboards, which this axis previously recorded it as showing.

• https://scale.com/genai-platform recorded 2026-08-13

  Automated and human feedback in the evaluation loop, SEAL as the in-house frontier benchmarking lab, and a continuous-learning flywheel over customer usage. Nothing externally reproducible, the benchmarks being private by design.