AlpacaEval
Tatsu Lab (Stanford)Automatic judge-based evaluator for instruction-following models: a model annotator compares a candidate's outputs against a reference on a fixed instruction set to produce a win rate and a leaderboard. AlpacaEval 2.0 adds a length-controlled win rate to remove the bias toward longer outputs, which raises correlation with Chatbot Arena.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public
- core features withheld
- no
Fully Apache-2.0. The README header carries a 'Data License CC By NC 4.0' badge, but both badges link to the sibling tatsu-lab/alpaca_farm repository rather than to anything in this one, so the NC term is not a license this product makes you accept.
- https://api.github.com/repos/tatsu-lab/alpaca_eval/license recorded 2026-09-22
GitHub's license endpoint reports spdx_id Apache-2.0 for tatsu-lab/alpaca_eval at path LICENSE, and returns the Apache-2.0 body. The repo's root listing carries no second license file.
- https://github.com/tatsu-lab/alpaca_eval recorded 2026-06-17
Apache-2.0, ~2.0k stars
- https://raw.githubusercontent.com/tatsu-lab/alpaca_eval/main/README.md recorded 2026-09-22
README installs the evaluator with `pip install alpaca-eval` or straight from the git repo and runs it against the reader's own model outputs, printing a leaderboard locally; the annotator configs and reference outputs ship in the package. The only external cost named is the reader's own OpenAI credits, roughly $10 a run. No paid tier, hosted edition or license key appears in it.
Adoption
2 medium confidenceAlpacaEval's only public adoption signal is its GitHub star count, since no download or install figure is published for the tool. Star counts alone cap how high a reading can go, because a star is not a use.
- https://api.github.com/repos/tatsu-lab/alpaca_eval recorded 2026-08-12
stargazers_count = 2,012 for tatsu-lab/alpaca_eval
Capability
4 medium confidenceAlpacaEval reports a length-controlled win rate that tracks Chatbot Arena rankings closely, with a public leaderboard and a toolkit for building and analyzing new automatic evaluators. It measures only judge-based instruction-following comparison, which is narrower than the multi-task harnesses it sits beside.
- https://github.com/tatsu-lab/alpaca_eval recorded 2026-06-17
AlpacaEval 2.0 LC win-rate
- https://pypi.org/pypi/alpaca-eval/json recorded 2026-09-29
The PyPI listing of the alpaca-eval package carries the project README, which says length-controlled win rates "are out and used by default! This increases the correlation with ChatBot Arena from 0.93 to 0.98" and lists a "Toolkit for building automatic evaluators" for building and analyzing new evaluators, alongside a leaderboard of common models.
- https://raw.githubusercontent.com/tatsu-lab/alpaca_eval/main/README.md recorded 2026-08-13
'Length-controlled Win Rates are out and used by default', raising correlation with Chatbot Arena from 0.93 to 0.98; LLM-as-judge auto-annotator against a reference baseline, plus a public leaderboard and tooling for building and analyzing new automatic evaluators.
- https://tatsu-lab.github.io/alpaca_eval/ recorded 2026-09-29
The leaderboard site is titled "AlpacaEval Leaderboard" and its caution line says "(LC) win rates alleviate length biases of GPT-4, but it may favor models finetuned on its outputs", above a ranked table of models.
Verified 2026-09-22