Model components / Evaluation code

AlpacaEval

Automatic judge-based evaluator for instruction-following models: a model annotator compares a candidate's outputs against a reference on a fixed instruction set to produce a win rate and a leaderboard. AlpacaEval 2.0 adds a length-controlled win rate to remove the bias toward longer outputs, which raises correlation with Chatbot Arena.

The README's data badge points at a sibling repository rather than anything in this one, which the openness axis resolves. Verified 2026-08-13 via the tatsu-lab/alpaca_eval repository, its license endpoint and its README.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated

Fully Apache-2.0. The README header carries a 'Data License CC By NC 4.0' badge, but both badges link to the sibling tatsu-lab/alpaca_farm repository rather than to anything in this one, so the NC term is not a license this product makes you accept.

https://github.com/tatsu-lab/alpaca_eval recorded 2026-06-17
Apache-2.0, ~2.0k stars
https://api.github.com/repos/tatsu-lab/alpaca_eval/license recorded 2026-08-13
GitHub's license endpoint reports spdx_id Apache-2.0 for tatsu-lab/alpaca_eval at path LICENSE, and returns the Apache-2.0 body. The repo's root listing carries no second license file.
https://raw.githubusercontent.com/tatsu-lab/alpaca_eval/main/README.md recorded 2026-08-13
README installs the evaluator with `pip install alpaca-eval` or straight from the git repo and runs it against the reader's own model outputs, printing a leaderboard locally; the annotator configs and reference outputs ship in the package. The only external cost named is the reader's own OpenAI credits, roughly $10 a run. No paid tier, hosted edition or license key appears in it.

Adoption

2 medium confidence

2.0

tatsu-lab/alpaca_eval carries 2,012 GitHub stars, which bands at 1K-10K stars, level 2. Stars cap a level at 3 in any case, because a star is not a use. No download, install or customer figure is published for this product, so stars are the only honest signal available and the level is directional.

https://api.github.com/repos/tatsu-lab/alpaca_eval recorded 2026-08-12
stargazers_count = 2,012 for tatsu-lab/alpaca_eval

Capability

4 medium confidence

4.0

The de-facto standard for its niche, with an influential length-controlled win-rate metric, but narrower in scope than the multi-task harnesses it sits beside.

https://github.com/tatsu-lab/alpaca_eval recorded 2026-06-17
AlpacaEval 2.0 LC win-rate
https://raw.githubusercontent.com/tatsu-lab/alpaca_eval/main/README.md recorded 2026-08-13
'Length-controlled Win Rates are out and used by default', raising correlation with Chatbot Arena from 0.93 to 0.98; LLM-as-judge auto-annotator against a reference baseline, plus a public leaderboard and tooling for building and analyzing new automatic evaluators.

Verified 2026-08-12