Model components / Benchmark / eval datasets

MT-Bench

Multi-turn open-ended benchmark of 80 questions for instruction-following and conversational ability, scored by a strong model acting as judge. Created by LMSYS, it ships inside FastChat with both the questions and GPT-4 reference answers published, and was introduced in the NeurIPS 2023 paper on judging with language models.

Small and partly saturated; a companion human-judgment dataset accompanies it. Verified 2026-08-12 via the FastChat llm_judge README, the FastChat LICENSE and the paper.

Openness

5 high confidence

5.0

questions: public(question.jsonl in the public FastChat git tree)
answers: public(GPT-4 references in FastChat)
license: Apache-2.0(FastChat repository LICENSE)
datasheet: present(fastchat/llm_judge/README.md documents the data, the judging protocol and the released judgment sets
arXiv: 2306.05685 documents composition)
access: ungated(questions and reference answers sit in the public git tree, no download barrier)

Both questions and reference answers are public and openly licensed. MT-Bench ships inside FastChat rather than as a standalone dataset repo, so its documentation is the llm_judge package README and the LLM-as-a-Judge paper rather than a Hugging Face card, and its access state is a public git tree rather than a Hub download.

https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge/data/mt_bench recorded 2026-06-17
public questions + GPT-4 references
https://raw.githubusercontent.com/lm-sys/FastChat/main/LICENSE recorded 2026-08-12
the FastChat repository, which contains the MT-Bench questions and reference answers, is licensed Apache License 2.0
https://raw.githubusercontent.com/lm-sys/FastChat/main/fastchat/llm_judge/README.md recorded 2026-08-12
"MT-bench is a set of challenging multi-turn open-ended questions for evaluating chat assistants"; documents the data files, the GPT-4 judging protocol, the pre-generated model answers and judgments, and the agreement computation. The reference answers and judgments are downloaded from the public repo by `download_mt_bench_pregenerated.py` with no credential or approval step

Adoption

2 medium confidence

2.0

2,361 downloads in the trailing 30 days for lmsys/mt_bench_human_judgments, which puts it in the 1K-10K band, level 2 on the dataset adoption scale. That scale tops out at >1M. The band is read off that package rather than off FastChat's 39.5k GitHub stars, which are shared across the whole platform: the dataset is the artifact being measured, and where a real download signal exists it governs over a star count.

https://huggingface.co/api/datasets/lmsys/mt_bench_human_judgments recorded 2026-08-12
2,361 downloads in the trailing 30 days for lmsys/mt_bench_human_judgments

Capability

4 medium confidence

4.0

The seminal multi-turn chat benchmark, and the one that pioneered LLM-as-judge scoring, though it is now small and partly saturated next to harder peers.

https://arxiv.org/abs/2306.05685 recorded 2026-08-13
"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"; the abstract still reads "we verify the agreement between LLM judges and human preferences by introducing two benchmarks - MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform", with GPT-4 judges reaching over 80% agreement with humans

Verified 2026-08-12