Artificial Analysis

platform

closed(proprietary benchmarking site + methodology)

source

partial(Stirrup covers two of the ten evals (AA-Briefcase, GDPval-AA)

methodology

documented(public methodology page, per-eval breakdown, temp/token params)

datasets

closed(maintains internal copies of all eval datasets)

harness

partial-open(Stirrup MIT, used for the agentic AA-Briefcase and GDPval-AA evals)

Artificial Analysis publishes its full methodology and releases Stirrup, an MIT-licensed agent harness that runs two of the Index's ten evaluations, as an open component. But the Index pipeline itself and its internal copies of every evaluation dataset are not published, which keeps it source-available rather than fully open source.

• https://api.github.com/repos/ArtificialAnalysis/Stirrup/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id MIT for ArtificialAnalysis/Stirrup - the one open component - and returns the MIT text as the LICENSE body. 546 stars.

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-09-29

  Methodology v4.3.2: "We maintain internal copies of all evaluation datasets" and "All models are run using our open source agentic harness" for agentic evals; the Index pipeline itself is not published. Searched the body for a pipeline repository: none.

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-09-29

  Artificial Analysis Intelligence Index v4.3.2 "incorporates 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, AA-LCR v1.1, AA-Omniscience, Humanity" and is a weighted average across four categories: "Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%)".

• https://github.com/ArtificialAnalysis/Stirrup recorded 2026-06-04

  Stirrup MIT-licensed agent harness, 413 stars (the only open component)

• https://pypi.org/pypi/stirrup/json recorded 2026-09-29

  PyPI metadata for the stirrup package (Artificial Analysis) carries the classifier "License :: OSI Approved :: MIT License" and a license text beginning "MIT License Copyright (c) 2025 Artificial Analysis".

The Intelligence Index is widely cited in frontier model coverage, and Artificial Analysis says it is publicly referenced by leading AI labs, publications and institutions. It publishes no unique-visitor or subscriber count, so this reflects citation reach rather than a usage figure.

• https://artificialanalysis.ai/ recorded 2026-06-04

  live Intelligence Index leaderboard positioned as the cross-model standard reference

• https://artificialanalysis.ai/ recorded 2026-09-29

  The live leaderboard headed "Independent analysis of AI" ranks models on Intelligence Index v4.3.2 with "28 of 679 models" shown. Searched the body for unique visitors, subscribers and monthly active users: none.

• https://artificialanalysis.ai/about recorded 2026-09-29

  The company describes itself as "the independent benchmarking company for AI" and says it is "Publicly referenced by the leading AI labs, publications, and institutions".

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-06-04

  independent standardized index cited as the reference methodology

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-09-29

  Methodology v4.3.2 describes an index that is "a helpful synthesis of overall language model intelligence" with standardized parameters and internal dataset copies. Searched the body for unique visitors, subscribers and monthly active users: none.

The Intelligence Index spans agentic, coding, general and scientific evaluations, run with standardized parameters on internal dataset copies that resist contamination. It is a curated methodology rather than an open harness, and because Artificial Analysis runs every evaluation privately, outsiders cannot reproduce the scores.

• https://artificialanalysis.ai/ recorded 2026-09-29

  The homepage leaderboard for Intelligence Index v4.3.2 shows a model list filtered as "28 of 679 models" and states "Independent analysis of AI".

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-09-29

  Methodology v4.3.2 lists 10 evaluations weighted "Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%)" and states "We maintain internal copies of all evaluation datasets" with a section of general testing parameters for all evals.

• https://artificialanalysis.ai/methodology/intelligence-benchmarking recorded 2026-09-29

  Artificial Analysis Intelligence Index v4.3.2 "incorporates 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, AA-LCR v1.1, AA-Omniscience, Humanity" and is a weighted average across four categories: "Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%)".