AI Potluck
Contribute
Back to Gap Map Model components / Evaluation code

Chatbot Arena

LMArena
source available / Overall score: 4.0(strong)

Hosted arena that ranks language models on blind pairwise human preference, turning crowd votes into an Elo-style leaderboard. Where a harness scores a model against fixed tasks, this scores it against another model as judged by whoever is at the keyboard. Formerly LMSYS Chatbot Arena and then LMArena, it now runs as Arena at arena.ai.

Openness

2 medium confidence
2.0
platform
closed(hosted arena.ai, proprietary live leaderboard + private vote data)
source
partial(FastChat is a real arena implementation but stale relative to the operating platform
reference-code
FastChat Apache-2.0 (powers Chatbot Arena but stale, last release Feb 2024, 39.5k stars)
aux-OSS
Arena-Hard-Auto Apache-2.0 (automatic LLM eval, highest correlation to LMArena)

Chatbot Arena runs as the hosted arena.ai platform, where the live ranking pipeline and vote data are closed. FastChat, its original Apache-2.0 implementation, has had no release since February 2024, and Arena-Hard-Auto is a separate Apache-2.0 automatic-eval companion, so the open code is real but is not the pipeline that runs the platform today.

Adoption

4 medium confidence
4.0

Chatbot Arena's reach is taken from the chat-request volume its FastChat README reports for the hosted platform. That figure predates the current arena.ai platform and has not been updated, so it is a dated, conservative measure rather than current usage.

  • https://arena.ai/ recorded 2026-06-04

    hosted live leaderboard / battle platform in active operation June 2026

  • https://arena.ai/ recorded 2026-09-29

    Hosted leaderboard platform titled "Arena AI: The Official AI Ranking" in active operation. Searched the body for a user count, traffic or request figure: none published on the page.

  • https://github.com/lm-sys/FastChat recorded 2026-06-04

    'serving over 10 million chat requests for 70+ LLMs'

  • https://raw.githubusercontent.com/lm-sys/FastChat/main/README.md recorded 2026-09-29

    README states "serving over 10 million chat requests for 70+ LLMs" and "Chatbot Arena has collected over 1.5M human votes from side-by-side LLM battles".

Capability

4 medium confidence
4.0

The hosted arena runs blind pairwise human-preference battles across many models, and Arena-Hard-Auto adds an automatic-judge companion built from hard real-world queries and creative-writing prompts. The evaluation code is largely closed and hosted, and it covers agentic and task-execution work less than harnesses built around running tasks.

Verified 2026-09-29