DeepEval

license

Apache-2.0(OSI, OSS framework)

source

public(GitHub)

commercial-tier

Confident AI managed eval/observability cloud on top

core features withheld

no — Confident AI is a separate hosted platform scored as its own product, the library evaluates locally without an account, no enterprise directory in the repo

DeepEval is an Apache-2.0 library whose metrics and test runs execute locally without an account. Confident AI, the hosted platform sold beside it and listed as its own product, does not gate the library, and no component is withheld from the published source.

• https://api.github.com/repos/confident-ai/deepeval/contents recorded 2026-08-11

  Repository root listing: deepeval, docs, examples, tests, typescript, scripts, skills, assets and packaging files. No ee/, enterprise/, or cloud/ directory under a second license. The repo's declared license is Apache-2.0.

• https://api.github.com/repos/confident-ai/deepeval/license recorded 2026-08-13

  GitHub's license endpoint reports spdx_id Apache-2.0 for confident-ai/deepeval and returns the Apache-2.0 text as the LICENSE body.

• https://github.com/confident-ai/deepeval recorded 2026-06-04

  Apache-2.0 license; Confident AI commercial cloud platform

• https://pypi.org/pypi/deepeval/json recorded 2026-09-29

  The PyPI record for release 4.2.6 carries a README ending in "DeepEval is licensed under Apache 2.0" with the classifier "License :: OSI Approved :: Apache Software License", sends contributors to "https://github.com/confident-ai/deepeval/blob/main/CONTRIBUTING.md", and says its metrics are powered by any "LLM of your choice, statistical methods, or NLP models that run" "locally on your machine"; the hosted Confident AI platform is presented as recommended, not required ("It is free, takes no additional code to setup, and we highly recommend giving it a try"). Searched the body for license key and paid: none.

• https://raw.githubusercontent.com/confident-ai/deepeval/main/README.md recorded 2026-08-11

  README states the metrics 'run locally on your machine', and directs readers to 'Sign up for Confident AI, the enterprise AI evals and observability platform' only to 'compare iterations, share evaluation reports, and monitor your AI in production'. The account section says using the platform 'will allow you to generate sharable testing reports on the cloud. It is free, takes no additional code to setup' - optional, and not a gate on any metric.

• https://raw.githubusercontent.com/confident-ai/deepeval/main/README.md recorded 2026-08-13

  README states the metrics 'run locally on your machine' and frames the Confident AI account as optional reporting that 'is free, takes no additional code to setup'. No metric is described as paid, licensed or enterprise-only.

DeepEval's adoption reading comes from its trailing 30-day PyPI downloads, the closest available usage signal for a library adopted directly into test suites. GitHub stars and the count of dependent projects point the same direction.

• https://github.com/confident-ai/deepeval recorded 2026-06-04

  15.9k stars; used by 1.3k projects

• https://pypistats.org/api/packages/deepeval/recent recorded 2026-08-13

  last_month = 6,218,151 downloads of the deepeval package (last_week 1,181,537, last_day 229,027)

• https://pypistats.org/packages/deepeval recorded 2026-06-04

  ~4,628,371 monthly PyPI downloads

DeepEval covers application, RAG, agentic and conversational evaluation with judge-model metrics such as G-Eval, and runs them as pytest tests. It sits a step below lm-evaluation-harness, which offers a far larger set of standardized academic benchmarks.

• https://github.com/confident-ai/deepeval recorded 2026-06-04

  G-Eval, RAG/agentic/conversational metrics, pytest-style eval

• https://pypi.org/pypi/deepeval/json recorded 2026-09-29

  The packaged README (release 4.2.6) lists G-Eval, DAG, Task Completion, Tool Correctness, "Answer Relevancy", Faithfulness, Hallucination, MCP Task Completion, multi-turn metrics such as "Knowledge Retention", and multimodal metrics, and says DeepEval is "similar to Pytest but specialized for unit testing LLM apps".

• https://pypi.org/pypi/deepeval/json recorded 2026-09-29

  The packaged README says users can benchmark any LLM on "MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, GSM8K" in under 10 lines of code, and can generate single and multi-turn synthetic datasets.

• https://raw.githubusercontent.com/confident-ai/deepeval/main/README.md recorded 2026-08-13

  README describes the framework as 'similar to Pytest but specialized for unit testing LLM apps' and documents G-Eval, Task Completion, Faithfulness, Hallucination, MCP Task Completion, conversational turn metrics and red-teaming.