Ragas is a Python-based tool for evaluating and optimizing LLM applications, with pre-built metrics, test data generation, and integrations for evaluation workflows. It has ongoing releases and a growing feature set since 2023-05-08.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Ragas is a Python-based toolkit for evaluating and optimizing Large Language Model applications. It provides objective metrics, test data generation, and integrations with common LLM frameworks and observability tools. It supports production-aligned test set generation and aims to enable data-driven improvements for LLM apps.
How it works
The repository aims to offer pre-built metrics for evaluating LLM outputs, generation of test datasets, and API-driven scoring mechanisms. It includes a collections API for metrics and data structures, as indicated by release notes mentioning migrations to a collections API and new methods for save/load in BasePrompt. The README shows an example using a DiscreteMetric to score response accuracy and an llm_factory-based LLM setup for scoring via a custom metric.
Getting started
Installation is through PyPI or from source:
pip install ragas
pip install git+https://github.com/vibrantlabsai/ragas
Quickstart commands include:
# List available templates
ragas quickstart
# Create a RAG evaluation project
ragas quickstart rag_eval
# Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project
The README provides an example demonstrating how to set up an LLM client and a custom DiscreteMetric for scoring a response.
Recent releases
Latest releases include:
- v0.4.3 (2026-01-13): added DSPyOptimizer with MIPROv2 for advanced prompt optimization; docs update for llms.txt generation.
- v0.4.2 (2025-12-23): migrate SQLSemanticEquivalence and DataCompyScore to collections API.
- v0.4.1 (2025-12-10): add save/load methods to BasePrompt; docs update for run_config collections API.
- v0.4.0 (2025-12-03): complete collections API documentation for remaining metrics; feature to support GPT-5 and o-series models.
- v0.3.9 (2025-11-11): docs fix and migration work related to SummaryScore.
Traction
Stars: 15122 (as of the data provided). Forks: 1600. Open issues: 535.
Behind the repo
No startup or company link is provided in the facts.
Caveats
License: Apache-2.0. Created: 2023-05-08. Last push: 2026-02-24. Language: Python. Topics include evaluation, llm, llmops. Open issues count is 535. No further licensing or age notes beyond these facts.