Artificial Analysis

platform

closed(proprietary benchmarking site + methodology)

source

partial(Stirrup covers one of ten evals

methodology

documented(public methodology page, per-eval breakdown, temp/token params)

datasets

closed(maintains internal copies of all eval datasets)

harness

partial-open(Stirrup MIT, used for the agentic GDPval-AA eval only, 413 stars)

Methodology is well documented and one component harness (Stirrup) is MIT, but the Index itself (pipeline, internal dataset copies, grader config) is proprietary. The earlier note reasoned "not source_available since the core isn't published", which reads source_available as a stronger claim than this ladder makes: source:partial covers a repo existing while the runtime does not ship, and Stirrup covers one of ten evals. Class corrected from open_core on 2026-07-30. In this ladder open_core means an OSI core with functionality withheld for a paid tier, and there is no open core here, only an open periphery. A repo that exists while the runtime does not ship is source:partial, which the ladder scores 2/source_available. The score itself was already 2.

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-06-04

  v4.0.4 (Mar 2026), 10-eval composite, internal copies of all datasets, standardized params (proprietary pipeline + open Stirrup harness for GDPval-AA)

• https://github.com/ArtificialAnalysis/Stirrup recorded 2026-06-04

  Stirrup MIT-licensed agent harness, 413 stars (the only open component)