Product / UX / Telemetry & observability

Evidently

Open-source framework to evaluate, test and monitor ML and LLM-powered systems, over both tabular and text data. Reports and Test Suites compute and gate on more than 100 built-in metrics, from data drift detection to LLM-as-judge scoring, and a Monitoring UI service visualizes those metrics and test results over time. The full monitoring dashboard is self-hostable; Evidently Cloud is a separate hosted product adding dataset and user management, alerting and no-code evals on top of the same open core.

Ruled into telemetry_observability rather than evaluation_code in the resolution ledger (2026-09-01, pull request 451 review) on product shape - continuous monitoring is Evidently's first-class surface and evaluation one workflow inside it, unlike a benchmark-runner or model-ranking harness. The evidently PyPI package is declared as a measurement artifact because installing it gives you the product - the evaluation library and the Monitoring UI service (`evidently ui`) both ship in that one package, its PyPI homepage points back at evidentlyai/evidently, and it is published by the project's own maintainers. Evidently Cloud is a separate hosted layer and is not what is scored here. Verified 2026-09-02 via the evidentlyai/evidently repository.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI, single root LICENSE)
source: public(the full Monitoring UI service is self-hostable from this repository, per the README's own "Self-host the open-source version" option)
core-gated: ungated(no ee/enterprise path in the recursive tree

Apache-2.0 throughout, with both the evaluation library and the monitoring dashboard published and self-hostable. The README states the choice directly - self-host the open-source Monitoring UI, or sign up for Evidently Cloud - and frames Cloud's extras (dataset/user management, alerting, no-code evals) as additions on top of the open core rather than functionality withheld from it. A recursive tree search found no enterprise, ee or license-key path.

https://raw.githubusercontent.com/evidentlyai/evidently/main/LICENSE recorded 2026-09-02
Apache License, Version 2.0 full text, the sole license file in the repository.
https://api.github.com/repos/evidentlyai/evidently/git/trees/main?recursive=1 recorded 2026-09-02
The whole repository tree, 1,528 paths. No ee/, enterprise or premium path exists; the only other LICENSE-named file is a vendored JS asset's own notice.
https://raw.githubusercontent.com/evidentlyai/evidently/main/README.md recorded 2026-09-02
"Monitoring UI service helps visualize metrics and test results over time. You can choose: Self-host the open-source version ... or Sign up for Evidently Cloud (Recommended). Evidently Cloud offers a generous free tier and extra features like dataset and user management, alerting, and no-code evals."

Adoption

4 medium confidence

4.0

The evidently package draws 1,255,790 PyPI downloads a month, inside the 1M-10M band but close to its floor, so a modest fall would put it at 3. The package is the product - the evaluation library and the self-hostable Monitoring UI service both install from it - which is why its count is the band rather than a proxy for one. 7,877 GitHub stars on a project active since 2020 corroborate without setting the level.

https://pypistats.org/api/packages/evidently/recent recorded 2026-09-02
last_month = 1,255,790 downloads for evidently (last_week 179,235, last_day 29,221)
https://pypi.org/pypi/evidently/json recorded 2026-09-02
info.name evidently, version 0.7.21, project_urls Homepage https://github.com/evidentlyai/evidently, author Emeli Dral, license Apache License 2.0 - the package points back at the declared repository and is published by the project's own maintainer.
https://api.github.com/repos/evidentlyai/evidently recorded 2026-09-02
stargazers_count = 7,877, forks_count = 906, created_at 2020-11-25, pushed_at 2026-08-31 (actively maintained)

Capability

3 high confidence

3.0

A deep and broad evaluation and monitoring library - more built-in metrics than any other product in this category - with continuous monitoring as its first-class surface, which is the product-shape ruling that placed it in this category. It has no prompt management and no playground, and its monitoring reads on metrics and test results computed over batches of data rather than live per-call tracing, which is the anchor's core surface. That combination sits one rung below Langfuse at 4, the same placement as openllmetry and monocle for a category-complete instrument missing prompt and playground breadth.

https://raw.githubusercontent.com/evidentlyai/evidently/main/README.md recorded 2026-09-02
Feature list: "100+ built-in metrics from data drift detection to LLM judges," Reports and Test Suites with pass/fail conditions, "Both offline evals and live monitoring," and a Monitoring Dashboard to "visualize metrics and test results over time" - no prompt-management or playground section anywhere in the document.

Verified 2026-09-02