Sierra

license

MIT(repository LICENSE body read on both repos

access

open(tasks and domain data ship in the repository, plain git clone, no gate)

answers

released(task definitions include the evaluation criteria

datasheet

yes(documents itself in the repository — domains README (structure, data format, available domains) plus arXiv:2506.07982

MIT covers the whole repository, with nothing held back, and the documentation lives in the repository itself rather than on a Hugging Face card, the same treatment this category already gives MT-Bench and LiveBench. Confidence is medium because the reading that no split is held out is inferred from the README and harness design rather than an explicit statement.

• https://raw.githubusercontent.com/sierra-research/tau-bench/main/LICENSE recorded 2026-10-06

  MIT License body, "Copyright (c) 2024 Sierra", for the τ¹ repo

• https://raw.githubusercontent.com/sierra-research/tau2-bench/main/LICENSE recorded 2026-10-06

  MIT License body, "Copyright (c) 2025 Sierra Research", standard text with no data carve-out

• https://raw.githubusercontent.com/sierra-research/tau2-bench/main/README.md recorded 2026-10-06

  title "τ-Bench:A Benchmark for Tool-Agent-User Interaction in Real-World Domains"; τ³-bench v1.0.0/v1.0.1 announcement; arXiv:2506.07982 badge; taubench.com leaderboard; git-clone install with data in-repo; domains README link documenting domain structure and data format; backward-compatibility note naming the base split as the original τ-bench task set

τ-bench has no package or dataset download count to measure: its only matching PyPI name belongs to an unrelated package, and the Hub carries no first-party dataset. GitHub stars on the sierra-research/tau2-bench and tau-bench repositories are used instead, though a star is not a use, and τ-bench's frequent citation as an agent benchmark in frontier model releases is not a download count either.

• https://api.github.com/repos/sierra-research/tau-bench recorded 2026-09-01

  stargazers_count 1414; license spdx MIT; description "Code and Data for Tau-Bench"

• https://api.github.com/repos/sierra-research/tau2-bench recorded 2026-09-01

  stargazers_count 1923; license spdx MIT; homepage taubench.com; pushed 2026-09-01

A dataset is not 'capable', so capability is left unscored, mirroring the category pattern (gsm8k). The task suite's difficulty is a quality property, not a capability score.

• https://raw.githubusercontent.com/sierra-research/tau2-bench/main/README.md recorded 2026-10-07

  a task suite plus harness; no performance, throughput or feature claim for the capability axis to read