Back to Gap Map Model components / Benchmark / eval datasets

τ-Bench

open / Overall score: 2.0

Benchmark for tool-using conversational agents that must coordinate with a simulated human user under domain policy, across airline, retail, telecom and banking-knowledge domains. Tasks, domain data and the evaluation harness ship together in the repository, with a public leaderboard at taubench.com. The current release, τ³-bench, adds voice and knowledge-retrieval evaluation on top of the τ² task set.

PyPI tau2 is an unrelated magnetic-relaxation package (gitlab.com/chilton-group/tau2), so no pypi artifact is declared; the Hub carries only third-party mirrors, so no huggingface_dataset either. One tier-level product covering the tau, tau2 and tau3 releases.

Openness

5 medium confidence

5.0

license: MIT(repository LICENSE body read on both repos
access: open(tasks and domain data ship in the repository, plain git clone, no gate)
answers: released(task definitions include the evaluation criteria
datasheet: yes(documents itself in the repository — domains README (structure, data format, available domains) plus arXiv:2506.07982

MIT covers the whole repository, with nothing held back, and the documentation lives in the repository itself rather than on a Hugging Face card, the same treatment this category already gives MT-Bench and LiveBench. Confidence is medium because the reading that no split is held out is inferred from the README and harness design rather than an explicit statement.

https://raw.githubusercontent.com/sierra-research/tau-bench/main/LICENSE recorded 2026-10-06
MIT License body, "Copyright (c) 2024 Sierra", for the τ¹ repo
https://raw.githubusercontent.com/sierra-research/tau2-bench/main/LICENSE recorded 2026-10-06
MIT License body, "Copyright (c) 2025 Sierra Research", standard text with no data carve-out
https://raw.githubusercontent.com/sierra-research/tau2-bench/main/README.md recorded 2026-10-06
title "τ-Bench:A Benchmark for Tool-Agent-User Interaction in Real-World Domains"; τ³-bench v1.0.0/v1.0.1 announcement; arXiv:2506.07982 badge; taubench.com leaderboard; git-clone install with data in-repo; domains README link documenting domain structure and data format; backward-compatibility note naming the base split as the original τ-bench task set

Adoption

2 medium confidence

2.0

τ-bench has no package or dataset download count to measure: its only matching PyPI name belongs to an unrelated package, and the Hub carries no first-party dataset. GitHub stars on the sierra-research/tau2-bench and tau-bench repositories are used instead, though a star is not a use, and τ-bench's frequent citation as an agent benchmark in frontier model releases is not a download count either.

https://api.github.com/repos/sierra-research/tau-bench recorded 2026-09-01
stargazers_count 1414; license spdx MIT; description "Code and Data for Tau-Bench"
https://api.github.com/repos/sierra-research/tau2-bench recorded 2026-09-01
stargazers_count 1923; license spdx MIT; homepage taubench.com; pushed 2026-09-01

Capability

not assessed

A dataset is not 'capable', so capability is left unscored, mirroring the category pattern (gsm8k). The task suite's difficulty is a quality property, not a capability score.

https://raw.githubusercontent.com/sierra-research/tau2-bench/main/README.md recorded 2026-10-07
a task suite plus harness; no performance, throughput or feature claim for the capability axis to read

Verified 2026-10-06, except some scores, still awaiting re-confirmation