AI Potluck
Model components / Evaluation code

Patronus Evaluation Platform

Patronus AI

Managed evaluation platform built around in-house judge models: Lynx for hallucination detection, GLIDER for explainable rubric judging and Percival for analyzing agent execution traces. It runs experiments for A/B testing, real-time production monitoring with tracing and alerts, and red-teaming, reached through Python and TypeScript SDKs.

The published SDKs wire to hosted infrastructure rather than being the engine, and the Python SDK repository carries no license file and has not been pushed since January 2026. Patronus publishes no customer or usage count. Verified 2026-08-13 via the Patronus documentation and the patronus-py repository.

Openness

2 medium confidence
2.0
platform
closed(hosted SaaS
source
partial(patronus-py is a client wiring to the hosted API, not the engine)
client-SDKs
open(patronus-py public on GitHub, v0.1.25, but only 7 stars and wires to the hosted API)

The open-source SDK is a thin client to a closed hosted evaluation backend: the core scoring engine and the managed Lynx and Glider evaluators are proprietary and cannot be self-hosted, and no license text is displayed on the SDK repo page. This is not an open core with functionality withheld for a paid tier; it is an open periphery around a runtime that never ships, which makes the source only partial and the product source-available rather than closed.

  • https://github.com/patronus-ai/patronus-py recorded 2026-06-04

    Python SDK whose remote evaluators 'run remotely on Patronus infrastructure'; v0.1.25 Jan 2026; 7 stars

  • https://docs.patronus.ai/docs recorded 2026-06-04

    hosted platform: built-in Lynx/Glider evaluators, experiments, production monitoring, SDK access to cloud service

  • https://raw.githubusercontent.com/patronus-ai/patronus-py/main/README.md recorded 2026-08-13

    SDK README's own example carries the comment '# This evaluator runs remotely on Patronus infrastructure.' - the published code wires to the hosted engine rather than being it.

  • https://api.github.com/repos/patronus-ai/patronus-py recorded 2026-08-13

    stargazers_count = 8, pushed_at 2026-01-09, and the repo's `license` field is null - GitHub detects no license file on the SDK at all.

  • https://docs.patronus.ai/docs recorded 2026-08-13

    An end-to-end hosted system for evaluating, monitoring and improving performance, with Lynx and Glider as in-house evaluators, experiments, real-time production monitoring and Python/TypeScript SDKs into the cloud service. No self-hostable or open-source evaluation engine is offered anywhere in it.

Adoption

not assessed

No user or customer counts and no usage volume appear anywhere on the docs or homepage. The open-source SDK carries 7 GitHub stars, which for a hosted platform is not an honest signal of anything - those stars belong to a thin client rather than to the product, and a star count would cap the level at hobbyist scale in any case. The docs name three case studies (Nova AI, Etsy and Weaviate) but attach no figures to them, so no level is assigned rather than inferring one from marketing.

Capability

3 medium confidence
3.0

A solid, broad commercial eval feature set - evaluators, experiments, monitoring and agent evaluation - but not frontier-defining within the category, and the closed backend limits reproducibility. Mid-tier on features overall.

  • https://docs.patronus.ai/docs recorded 2026-06-04

    Lynx/Glider evaluators, RAG/agent/NLP/OWASP metrics, experiments, red-teaming, monitoring

  • https://docs.patronus.ai/docs recorded 2026-08-13

    Lynx and Glider in-house evaluators plus custom ones, experiments for A/B testing, real-time production monitoring through tracing and alerts, and Python/TypeScript SDKs. Agent evaluation and Percival present. Nothing independently reproducible, the engine being closed.

Verified 2026-08-13