AI Potluck
Model components / Evaluation code

promptfoo

promptfoo

Declarative CLI and Node library that tests prompts, agents and RAG pipelines from a YAML config, comparing model outputs side by side and running the same assertions in CI. It also ships a red-teaming mode that scans an application for jailbreaks and other vulnerabilities, which most eval harnesses leave out.

Promptfoo became part of OpenAI and the project continues in the open. It is distributed npm-first, so the PyPI package is a minority channel and is deliberately not declared as an artifact. Verified 2026-08-13 via the promptfoo repository, its license endpoint and its README.

Openness

5 high confidence
5.0
license
MIT(OSI)
source
public(GitHub)
core-gated
ungated

MIT-licensed and fully public, and MIT is an OSI license, so this is open source. OpenAI now owns the project, which does not change the license the code ships under.

Adoption

4 high confidence
4.0

The npm registry reports 2,277,977 downloads of promptfoo in the trailing month, inside the 1M-10M band, against 21.9k GitHub stars, and the project states it powers LLM apps serving more than 10M users in production. npm is the right channel to read because promptfoo is an npm-first CLI. A PyPI package named promptfoo also exists and is genuinely this project's, but it sees only 13,629 downloads a month; banding on that would substitute a minority channel for the product's real distribution and understate its reach by two rungs.

Capability

4 high confidence
4.0

Very broad eval coverage plus a distinctive red-teaming and security axis. Top-tier for application evaluation, though below the academic-benchmark standardization references such as lm-evaluation-harness on raw benchmark coverage.

Verified 2026-08-13