Back to Gap Map Model components / Evaluation code

YYLO Benchmark

open source / Overall score: 1.5

Benchmark harness for coding agents that runs every candidate attempt in a dedicated fresh repository with private home, temp, cache, and config roots behind a selective filesystem sandbox, and retains one hash-verified evidence chain linking the plan, attempt, workspace receipt, post-execution repository manifest, terminal, and evaluation IDs. Recovery, doctor, and report fail closed unless the complete chain verifies. It offers isolated v2 attempts plus an explicitly authorized governed-workflow lane, and distributes as the @yylo/benchmark npm package.

Verified 2026-09-14 via the yylo-dev/yylo-benchmark repository, its LICENSE, and the npm registry download API.

Openness

5 high confidence

5.0

license: MIT(OSI)
source: public
core-gated: ungated

Verified MIT (standard license text, Copyright (c) 2026 JUNO AI INC.), full harness source public on GitHub, no paid tier, managed offering, or enterprise directory. Paid and provider execution always requires separate consumer authorization; package tests use synthetic boundaries only.

https://github.com/yylo-dev/yylo-benchmark/blob/main/LICENSE recorded 2026-09-14
MIT License text, Copyright (c) 2026 JUNO AI INC., standard OSI license body
https://github.com/yylo-dev/yylo-benchmark recorded 2026-09-14
MIT license on the repo, full harness source public, 1 star; no managed tier, no paid plan, no enterprise directory

Adoption

1 medium confidence

1.0

The npm download API returns 326 downloads for @yylo/benchmark over the trailing month, which bands at level 1 (<10K monthly) on the software scale. @yylo/benchmark is the product’s declared and only distribution channel, so the figure measures the product itself rather than standing in for it. Raw download volume includes CI and mirror traffic, so this is install volume rather than a user count; an early-stage harness, included as the open evaluation layer of a category whose marquee harnesses are framework-scale projects.

https://api.npmjs.org/downloads/point/last-month/@yylo/benchmark recorded 2026-09-14
npm registry download API returns 326 downloads for @yylo/benchmark over the trailing month, 2026-08-13 to 2026-09-11

Capability

2 medium confidence

2.0

Narrow, early evaluation harness (326 npm installs a month, 1 star); capable within its scope - isolated attempt workspaces, retained hash-verified evidence chains, fail-closed doctor and report - but young and with no third-party adoption, so scored 2. No peer comparison is recorded because the scale bottoms out before the anchor is reachable.

https://raw.githubusercontent.com/yylo-dev/yylo-benchmark/main/README.md recorded 2026-09-14
README describing two evaluation lanes, isolated fresh-repository attempt workspaces with selective filesystem sandboxing, deterministic and LLM-judge evaluator profiles, and fail-closed recovery, doctor, and report over one retained hash-verified evidence chain

Verified 2026-09-14