Back to Gap Map Model components / Benchmark / eval datasets

AgentDojo

open / Overall score: 3.0

Benchmark for evaluating prompt-injection attacks and defenses on tool-using LLM agents. It ships realistic task suites across workspace, banking, travel and Slack environments together with injection test cases, and grades agents on both task completion and security. The suite installs from PyPI and publishes results for common models and defenses.

Boundary call: a prompt-injection benchmark filed as benchmark data rather than a safeguard, because the product is an evaluation suite.

Openness

5 high confidence

5.0

license: MIT(repository LICENSE body read
access: open(pip install agentdojo
answers: released(ground-truth task and security checks ship with the suite and grade locally
datasheet: present(documentation site agentdojo.spylab.ai with API and suite docs, published results pages, plus arXiv:2406.13352)

MIT covers the whole repository, and both the suite and its evaluation code are distributed openly through PyPI and GitHub. The maintained documentation site serves in place of a Hugging Face dataset card, which this category accepts from a repository-hosted benchmark.

https://pypi.org/pypi/agentdojo/json recorded 2026-10-06
version 0.1.35; project_urls.repository = github.com/ethz-spylab/agentdojo, homepage agentdojo.spylab.ai — the backlink that verifies package identity
https://raw.githubusercontent.com/ethz-spylab/agentdojo/main/LICENSE recorded 2026-10-06
MIT License body, copyright the six authors (Debenedetti, Zhang, Balunovic, Beurer-Kellner, Fischer, Tramèr), standard text with no data carve-out
https://raw.githubusercontent.com/ethz-spylab/agentdojo/main/README.md recorded 2026-10-06
"A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents"; pip install agentdojo quickstart; paper link arXiv:2406.13352; results at agentdojo.spylab.ai/results; ETH Zurich + Invariant Labs affiliation

Adoption

3 medium confidence

3.0

Adoption is measured by PyPI downloads of the agentdojo package, which ships the full benchmark suite and whose PyPI metadata links back to the same repository. This category usually reads adoption from Hugging Face dataset downloads, but since no Hugging Face dataset exists here, the package's own downloads are read as the usage signal instead, the same treatment used for evalplus.

https://pypistats.org/api/packages/agentdojo/recent recorded 2026-09-01
last_month 24,142 downloads for agentdojo

Capability

not assessed

A dataset is not 'capable', so capability is left unscored, mirroring the category pattern (gsm8k).

https://raw.githubusercontent.com/ethz-spylab/agentdojo/main/README.md recorded 2026-10-07
an evaluation suite; no performance, throughput or feature claim for the capability axis to read

Verified 2026-09-24