Back to Gap Map Model components / Evaluation code

SimpleAudit

Simula Research Laboratory
open source / Overall score: 2.0

Local-first framework for multilingual auditing and red-teaming of AI systems through adversarial probing, developed by Simula and SimulaMet in Norway with the Norwegian Directorate of Health. It runs open models locally so prompts, transcripts and policies stay inside the deployment environment, and ships scenario packs for safety, RAG, healthcare and broken-premise testing. It is a verified Digital Public Good.

A fixed default scenario pack, rubric, auditor, judge and sampling configuration make reruns comparable.

Openness

5 high confidence
5.0
license
MIT(OSI
source
public
core features withheld
no — local execution is the default mode, all scenario packs ship in the package

SimpleAudit is MIT-licensed with full public source, and MIT is an OSI license. The README describes a local-first framework that ships its scenario packs and needs no account, and no license key, subscription or pricing term appears in it.

Adoption

1 medium confidence
1.0

GitHub stars are the only signal published for SimpleAudit, since no download, install or customer figure exists. A star is not a use, so this reading stays directional rather than precise.

Capability

3 high confidence
3.0

The README reports responsiveness of AUROC 0.89-1.00 across reliable judge-auditor cells and compares the tool against Petri. It is a single-purpose adversarial-probing framework rather than a general harness, with broken-premise packs beside its safety, RAG and health packs.

  • https://github.com/kelkalot/simpleaudit recorded 2026-06-22

    README: adversarial probing, local-first, scenario packs, broken-premise runner

  • https://raw.githubusercontent.com/kelkalot/simpleaudit/main/README.md recorded 2026-08-13

    multilingual adversarial probing with local open models or optional API-hosted ones; a fixed default scenario pack, rubric, auditor, judge and sampling configuration so reruns are comparable; reported responsiveness of AUROC 0.89-1.00 across judge-auditor cells; results reported as a bundle rather than a leaderboard rank.

  • https://raw.githubusercontent.com/kelkalot/simpleaudit/main/README.md recorded 2026-09-29

    README reports responsiveness of "AUROC 0.89–1.00 across reliable judge–auditor cells", compares against Petri ("both tools pass, so the differences live upstream of the chain"), lists broken-premise packs "health_bullshit" and "epistemic_safety" beside the safety, rag and health packs, and advises "Report the bundle, not a leaderboard."

Verified 2026-09-27