Back to Gap Map Model components / Benchmark / eval datasets

MT-Bench

open / Overall score: 2.0

Multi-turn open-ended benchmark of 80 questions for instruction-following and conversational ability, scored by a strong model acting as judge. Created by LMSYS, it ships inside FastChat with both the questions and GPT-4 reference answers published, and was introduced in the NeurIPS 2023 paper on judging with language models.

Openness

5 high confidence

5.0

questions: public(question.jsonl in the public FastChat git tree)
answers: public(GPT-4 references in FastChat)
license: Apache-2.0(FastChat repository LICENSE)
datasheet: present(fastchat/llm_judge/README.md documents the data, the judging protocol and the released judgment sets
arXiv: 2306.05685 documents composition)
access: ungated(questions and reference answers sit in the public git tree, no download barrier)

Both questions and reference answers are public and openly licensed. MT-Bench ships inside FastChat rather than as a standalone dataset repo, so its documentation is the llm_judge package README and the LLM-as-a-Judge paper rather than a Hugging Face card, and its access state is a public git tree rather than a Hub download.

https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge/data/mt_bench recorded 2026-06-17
public questions + GPT-4 references
https://raw.githubusercontent.com/lm-sys/FastChat/main/fastchat/llm_judge/README.md recorded 2026-10-13
"MT-bench is a set of challenging multi-turn open-ended questions for evaluating chat assistants"; documents the data files, the GPT-4 judging protocol, the pre-generated model answers and judgments, and the agreement computation. The reference answers and judgments are downloaded from the public repo by `download_mt_bench_pregenerated.py` with no credential or approval step
https://raw.githubusercontent.com/lm-sys/FastChat/main/LICENSE recorded 2026-10-13
the FastChat repository, which contains the MT-Bench questions and reference answers, is licensed Apache License 2.0

Adoption

2 medium confidence

2.0

Adoption is measured through Hugging Face downloads of lmsys/mt_bench_human_judgments, the companion dataset carrying MT-Bench's human judgments, rather than through FastChat's GitHub stars, which cover the whole codebase rather than this benchmark specifically. A download count that measures the actual dataset is read ahead of a star count that does not.

https://huggingface.co/api/datasets/lmsys/mt_bench_human_judgments recorded 2026-08-12
2,361 downloads in the trailing 30 days for lmsys/mt_bench_human_judgments

Capability

not assessed

A dataset is not 'capable' the way software or a model is, so capability is left unscored. MT-Bench is 80 multi-turn questions across eight categories, scored by an LLM acting as judge; it introduced LLM-as-judge scoring but is now small and partly saturated next to harder peers.

https://arxiv.org/abs/2306.05685 recorded 2026-09-29
Abstract page titled "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" describing "MT-bench, a multi-turn question set" and "Chatbot Arena, a crowdsourced battle platform", with GPT-4 judges reaching over 80% agreement with humans. Searched the body for leaderboard, SOTA, state-of-the-art and eight categories: none.

Verified 2026-09-27