RewardBench
Allen Institute for AIBenchmark for evaluating reward models, pairing prompts with chosen and rejected responses across chat, reasoning and safety subsets. RewardBench 2 extends it with harder, unseen human prompts, and both versions publish their datasets, results and a public leaderboard. The repository provides a common inference format for scoring most open reward models.
One product covering RewardBench and RewardBench 2; the repo README fronts both with datasets, results and leaderboard links. PyPI rewardbench's home_page points at a 404 path, so no pypi artifact is declared. Verified 2026-09-01 via the allenai/reward-bench GitHub API record, the LICENSE body, the repository README and both Hub dataset cards.
Openness
5 high confidence- license
- ODC-BY(license odc-by on both Hub dataset cards
- access
- open(both Hub datasets ungated ("gated":false), downloadable parquet)
- answers
- released(chosen/rejected pairs are the labels and ship in the public splits
- datasheet
- present(dataset cards on both Hub releases with schema, subsets and citation)
ODC-BY on both dataset cards, ungated downloads, cards present. Ladder walk: license_tier open_data (odc-by is an enumerated open_data spelling) + documentation present → 5/open. Governing release is RewardBench 2, same license as v1, so the combine rule moves nothing.
- https://huggingface.co/datasets/allenai/reward-bench/raw/main/README.md recorded 2026-09-01
card front matter license: odc-by; features prompt/chosen/chosen_model/rejected/ rejected_model; subsets and filtering documented
- https://huggingface.co/api/datasets/allenai/reward-bench recorded 2026-09-01
"gated":false; tags license:odc-by; 10,022 downloads in the trailing 30 days
- https://huggingface.co/api/datasets/allenai/reward-bench-2 recorded 2026-09-01
"gated":false; tags license:odc-by; 4,212 downloads in the trailing 30 days
- https://raw.githubusercontent.com/allenai/reward-bench/main/LICENSE recorded 2026-09-01
Apache License 2.0 full body, covering the evaluation code
- https://raw.githubusercontent.com/allenai/reward-bench/main/README.md recorded 2026-09-01
fronts RewardBench and RewardBench 2 together — leaderboard space, both eval datasets, both results sets, papers arXiv:2403.13787 and arXiv:2506.01937
Adoption
3 high confidence14,234 Hugging Face downloads in the trailing 30 days summed across the two declared datasets (allenai/reward-bench 10,022 + allenai/reward-bench-2 4,212), the 10K-100K band, level 3 on the dataset scale. RewardBench is the standard reward-model leaderboard, but the band rests on the measured figure.
- https://huggingface.co/api/datasets/allenai/reward-bench recorded 2026-09-01
10,022 downloads in the trailing 30 days
- https://huggingface.co/api/datasets/allenai/reward-bench-2 recorded 2026-09-01
4,212 downloads in the trailing 30 days
Capability
not assessedA dataset is not 'capable', so this axis is left unscored, mirroring the category pattern (gsm8k).
- https://huggingface.co/datasets/allenai/reward-bench/raw/main/README.md recorded 2026-09-01
a static preference-pair corpus; no performance, throughput or feature claim for the capability axis to read
Verified 2026-09-01