Model components / Evaluation code

Ragas

Evaluation framework purpose-built for Retrieval-Augmented Generation pipelines with 30+ metrics covering retrieval quality (context precision/recall), generation faithfulness, and answer relevance. Picked when the system under test is RAG; Ragas decomposes pipeline failure into retrieval vs. generation buckets in a way general harnesses do not. 14k stars, 240 contributors.

explodinggradients/ragas, v0.4.3 (released 13 Jan 2026), Apache-2.0, ~14.2k stars. Toolkit for evaluating LLM/RAG applications: LLM-based + traditional metrics, synthetic test-set generation, LangChain/observability integrations. Confirmed live on GitHub + PyPI June 2026. Note: maintainer Exploding Gradients also runs a commercial app.ragas.io cloud tier (open-core), though the core library is fully OSI.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public(full metrics + test-gen library)
core-gated: ungated

Core library is fully OSI (Apache-2.0) with no feature-gated functionality; could be argued open_core given the hosted cloud product, but the eval library itself is complete and open, so scored open_source.

https://github.com/explodinggradients/ragas recorded 2026-06-04
Apache-2.0 license; LLM-app eval metrics + test-set generation; v0.4.3 (Jan 2026); 14.2k stars

Adoption

4 high confidence

4.0

~1.51M PyPI downloads in the last 30 days (pypistats), de facto standard for RAG/LLM-app evaluation, widely integrated (LangChain, LlamaIndex, observability platforms). Stars (14.2k) corroborate. Solidly in the 1M-10M monthly band.

https://pypistats.org/packages/ragas recorded 2026-06-04
1,513,553 downloads in the last month
https://github.com/explodinggradients/ragas recorded 2026-06-04
14.2k stars; LangChain/observability integrations

Capability

4 medium confidence

4.0

Frontier within the application/RAG-eval niche (metric breadth + test generation), but not a general academic-benchmark model harness; scored 4 not 5 because coverage is domain-scoped.

https://github.com/explodinggradients/ragas recorded 2026-06-04
objective metrics (LLM-based + traditional), Aspect Critique, automatic test-data generation, integrations

Unchanged since 2026-07-30 (last edited, not re-checked)