Evidently
Evidently AIOpen-source framework to evaluate, test and monitor ML and LLM-powered systems, over both tabular and text data. Reports and Test Suites compute and gate on more than 100 built-in metrics, from data drift detection to LLM-as-judge scoring, and a Monitoring UI service visualizes those metrics and test results over time. The full monitoring dashboard is self-hostable; Evidently Cloud is a separate hosted product adding dataset and user management, alerting and no-code evals on top of the same open core.
Ruled into telemetry_observability rather than evaluation_code in the resolution ledger (2026-09-01, pull request 451 review) on product shape - continuous monitoring is Evidently's first-class surface and evaluation one workflow inside it, unlike a benchmark-runner or model-ranking harness. The evidently PyPI package is declared as a measurement artifact because installing it gives you the product - the evaluation library and the Monitoring UI service (`evidently ui`) both ship in that one package, its PyPI homepage points back at evidentlyai/evidently, and it is published by the project's own maintainers. Evidently Cloud is a separate hosted layer and is not what is scored here.
Openness
5 high confidence- license
- Apache-2.0(OSI, single root LICENSE)
- source
- public(the full Monitoring UI service is self-hostable from this repository, per the README's own "Self-host the open-source version" option)
- core features withheld
- no — no ee/enterprise path in the recursive tree
Evidently is released under Apache-2.0 throughout, and both the evaluation library and the self-hostable monitoring dashboard are published in the same repository. Evidently Cloud adds dataset and user management, alerting and no-code evals on top of that open core, rather than withholding any of it.
- https://api.github.com/repos/evidentlyai/evidently/git/trees/main?recursive=1 recorded 2026-09-02
The whole repository tree, 1,528 paths. No ee/, enterprise or premium path exists; the only other LICENSE-named file is a vendored JS asset's own notice.
- https://raw.githubusercontent.com/evidentlyai/evidently/main/LICENSE recorded 2026-10-07
Apache License, Version 2.0 full text, the sole license file in the repository.
- https://raw.githubusercontent.com/evidentlyai/evidently/main/README.md recorded 2026-10-07
"Monitoring UI service helps visualize metrics and test results over time. You can choose: Self-host the open-source version ... or Sign up for Evidently Cloud (Recommended). Evidently Cloud offers a generous free tier and extra features like dataset and user management, alerting, and no-code evals."
Adoption
3 medium confidenceAdoption is measured by PyPI downloads of the evidently package, since installing it delivers the whole product: the evaluation library and the self-hostable Monitoring UI both ship from the same package, so the count is a direct measure of use rather than a proxy. GitHub stars on the project, active since 2020, corroborate without adding to the measurement.
- https://api.github.com/repos/evidentlyai/evidently recorded 2026-09-02
stargazers_count = 7,877, forks_count = 906, created_at 2020-11-25, pushed_at 2026-08-31 (actively maintained)
- https://pypi.org/pypi/evidently/json recorded 2026-09-02
info.name evidently, version 0.7.21, project_urls Homepage https://github.com/evidentlyai/evidently, author Emeli Dral, license Apache License 2.0 - the package points back at the declared repository and is published by the project's own maintainer.
- https://pypistats.org/api/packages/evidently/recent recorded 2026-10-07
823,148 downloads in the trailing 30 days for evidently (pypistats last_month)
Capability
3 high confidenceEvidently is a deep evaluation and monitoring library, with an unusually large set of built-in metrics and continuous monitoring as its main surface. Its monitoring reads metrics and test results computed over batches of data rather than live per-call tracing, which keeps it a step behind Langfuse.
- https://raw.githubusercontent.com/evidentlyai/evidently/main/README.md recorded 2026-10-07
Feature list: "100+ built-in metrics from data drift detection to LLM judges," Reports and Test Suites with pass/fail conditions, "Both offline evals and live monitoring," and a Monitoring Dashboard to "visualize metrics and test results over time" - no prompt-management or playground section anywhere in the document.
Verified 2026-10-07, except some scores, still awaiting re-confirmation