AI Potluck
Model components / Evaluation code

Artificial Analysis Intelligence Index

Artificial Analysis

Independent benchmarking service that runs 10+ academic and agentic evals (MMLU, GPQA, long-context, etc.) against every major model and publishes the Intelligence Index, a composite score with 95% CIs. Picked as the go-to neutral leaderboard for model selection: their Index v3 score is widely cited and covers 300+ models across 100+ providers. Raised a seed round via the Nat Friedman / Daniel Gross AIGrant program.

Artificial Analysis Intelligence Index v4.0.4 (March 2026). Composite of 10 evals (GDPval-AA, tau2-Bench Telecom, Terminal-Bench Hard, SciCode, AA-LCR, AA-Omniscience, IFBench, Humanity's Last Exam, GPQA Diamond, CritPt) in 4 equal-weighted categories. Independent third-party benchmarking platform with published methodology; partial OSS (Stirrup agent harness, MIT, used for GDPval-AA) but the Index pipeline + internal dataset copies are proprietary. Confirmed live June 2026.

Openness

2 high confidence
2.0
platform
closed(proprietary benchmarking site + methodology)
source
partial(Stirrup covers one of ten evals
methodology
documented(public methodology page, per-eval breakdown, temp/token params)
datasets
closed(maintains internal copies of all eval datasets)
harness
partial-open(Stirrup MIT, used for the agentic GDPval-AA eval only, 413 stars)

Methodology is well documented and one component harness (Stirrup) is MIT, but the Index itself (pipeline, internal dataset copies, grader config) is proprietary. The earlier note reasoned "not source_available since the core isn't published", which reads source_available as a stronger claim than this ladder makes: source:partial covers a repo existing while the runtime does not ship, and Stirrup covers one of ten evals. Class corrected from open_core on 2026-07-30. In this ladder open_core means an OSI core with functionality withheld for a paid tier, and there is no open core here, only an open periphery. A repo that exists while the runtime does not ship is source:partial, which the ladder scores 2/source_available. The score itself was already 2.

Adoption

4 medium confidence
4.0

The AA Intelligence Index is one of the most-cited independent model-ranking standards in frontier release coverage and the flagship_scores set itself uses it as a capability basis (e.g. Grok 4.20, Gemini 3.5 Flash records). Widely referenced across vendor launches and tech press as the go-to cross-model index. No disclosed unique-visitor count; level 4 reflects its standard-citation reach across the industry, banded conservatively. Directional.

Capability

4 high confidence
4.0

Frontier-grade composite coverage spanning agentic/coding/general/scientific with contamination-resistant internal datasets and standardized running. Strong capability. Not a 5: it is a curated index/methodology rather than an open community harness, and held-out/proprietary running limits external reproducibility.

Unchanged since 2026-07-30 (last edited, not re-checked)