ETH Zurich SPY Lab

license

MIT(repository LICENSE body read

access

open(pip install agentdojo

answers

released(ground-truth task and security checks ship with the suite and grade locally

datasheet

present(documentation site agentdojo.spylab.ai with API and suite docs, published results pages, plus arXiv:2406.13352)

MIT covers the whole repository, and both the suite and its evaluation code are distributed openly through PyPI and GitHub. The maintained documentation site serves in place of a Hugging Face dataset card, which this category accepts from a repository-hosted benchmark.

• https://pypi.org/pypi/agentdojo/json recorded 2026-10-06

  version 0.1.35; project_urls.repository = github.com/ethz-spylab/agentdojo, homepage agentdojo.spylab.ai — the backlink that verifies package identity

• https://raw.githubusercontent.com/ethz-spylab/agentdojo/main/LICENSE recorded 2026-10-06

  MIT License body, copyright the six authors (Debenedetti, Zhang, Balunovic, Beurer-Kellner, Fischer, Tramèr), standard text with no data carve-out

• https://raw.githubusercontent.com/ethz-spylab/agentdojo/main/README.md recorded 2026-10-06

  "A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents"; pip install agentdojo quickstart; paper link arXiv:2406.13352; results at agentdojo.spylab.ai/results; ETH Zurich + Invariant Labs affiliation

Adoption is measured by PyPI downloads of the agentdojo package, which ships the full benchmark suite and whose PyPI metadata links back to the same repository. This category usually reads adoption from Hugging Face dataset downloads, but since no Hugging Face dataset exists here, the package's own downloads are read as the usage signal instead, the same treatment used for evalplus.

• https://pypistats.org/api/packages/agentdojo/recent recorded 2026-09-01

  last_month 24,142 downloads for agentdojo

A dataset is not 'capable', so capability is left unscored, mirroring the category pattern (gsm8k).

• https://raw.githubusercontent.com/ethz-spylab/agentdojo/main/README.md recorded 2026-10-07

  an evaluation suite; no performance, throughput or feature claim for the capability axis to read