Introduction
Evidently is an open-source Python framework designed for evaluating, testing, and monitoring machine learning (ML) and large language model (LLM) systems. Developed by Evidently AI and released under the Apache 2.0 license, the framework supports both offline evaluations and live monitoring. It integrates over 100 built-in metrics covering data quality, model performance, and LLM output evaluation, making it a versatile tool for AI observability across the development lifecycle.
The framework is modular, allowing users to start with one-off evaluations or deploy a full monitoring service. It works with tabular and text data, supporting predictive and generative tasks such as classification, regression, ranking, and retrieval-augmented generation (RAG).
Evidently is available as a standalone Python library and as part of a self-hosted platform, with an optional cloud-based version (Evidently Cloud) offering additional features like dataset management, alerting, and no-code workflows.
How It Works
Evidently operates through three primary components: Reports, Test Suites, and a Monitoring Dashboard.
Reports compute and summarize evaluations using built-in metrics or custom configurations. Users can generate reports for data quality, drift detection, model performance, and LLM outputs. These reports can be viewed interactively in Python or exported as JSON, Python dictionaries, or HTML. For example, a Data Drift Preset can detect shifts in column distributions using statistical tests like Population Stability Index (PSI), while a TextEvals preset evaluates sentiment, length, and toxicity in LLM-generated text.
Test Suites extend Reports by adding pass/fail conditions, enabling automated regression testing, CI/CD checks, or data validation. Users can define thresholds (e.g., greater than, less than) or auto-generate conditions from reference datasets. This feature ensures consistency in AI system performance over time, reducing the risk of undetected degradation.
The Monitoring Dashboard provides a visual interface to track metrics and test results over time. Users can self-host the open-source version or use Evidently Cloud for managed services, which includes additional features like dataset storage, user management, and alerting. The dashboard supports both batch evaluations and real-time monitoring, making it suitable for production environments.
Evidently’s architecture is open and modular, allowing users to export data and integrate with other tools. It supports Python interfaces for custom metrics and works with popular libraries like Pandas and Scikit-learn. The framework also includes utilities for synthetic data generation and prompt optimization, enhancing its utility for LLM development and testing.
Use Cases
Evidently is used across a range of AI applications, from model experimentation to production monitoring.
- Data Quality and Drift Detection: Evaluating tabular data for missing values, duplicates, outliers, and distribution shifts.
- LLM Output Evaluation: Assessing generative outputs for semantic similarity, retrieval relevance, summarization quality, and adherence to prompts. Evidently supports LLM-as-a-judge evaluations, where another LLM evaluates the quality of generated responses, and provides metrics for ranking tasks (NDCG, MAP, MRR) and recommendation systems (serendipity, diversity).
- Regression Testing for LLMs: Systematic checks for changes in LLM outputs, such as shifts in tone, content, or length. This is particularly useful in CI/CD pipelines, where Evidently can be integrated with GitHub Actions to auto-test LLM systems on every code update.
- RAG System Evaluation: Scoring context relevance, generation quality, and evidence usage in retrieval-augmented generation systems. Evidently provides metrics to measure how well LLMs incorporate retrieved information into their responses.
- Prompt Optimization: Automating the generation and tuning of prompts for LLM judges. The framework includes tools to align LLM judges with human-labeled datasets, improving the reliability of automated evaluations.
Evidently’s flexibility extends to synthetic data generation, enabling users to create custom test datasets that mimic real-world scenarios. This is particularly valuable for stress-testing AI systems under various conditions.
Key Features and Capabilities
Evidently provides a comprehensive set of tools for AI observability, with over 100 built-in metrics categorized into distinct evaluation areas:
- Text Descriptors: Metrics for analyzing text data, including length, sentiment, toxicity, language detection, and regular expression matches. These are useful for evaluating LLM outputs and chatbot responses.
- These ensure the integrity of training and inference datasets.
- This is critical for maintaining model reliability in non-stationary environments.
- Ranking Metrics: Metrics for information retrieval and recommendation systems, including Normalized Discounted Cumulative Gain (NDCG), Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), and Hit Rate.
- Evidently supports LLM-as-a-judge evaluations, where another LLM assesses the quality of generated content.
- Custom Metrics: Users can define their own metrics using Python, extending Evidently’s functionality to domain-specific requirements.
- Embeddings Drift Metric: A new metric to detect drift in ML embeddings, which is essential for models relying on vector representations of data.
- Open-Source Tracing and Dataset Management: Features previously available only in Evidently Cloud, such as LLM tracing storage and raw dataset management, are now part of the open-source version.
- Prompt Optimization Tools: Automated prompt generation and tuning for LLM judges, reducing the manual effort required to design effective evaluation prompts.
Evidently’s API has also undergone updates to improve flexibility and ease of use. The framework supports both offline evaluations (batch processing) and live monitoring, making it suitable for production environments where real-time observability is required.
Technical Implementation and Updates
Evidently is implemented in Python and can be installed via PyPI or Conda. Key technical details include:
- Installation: Users can install Evidently using pip install evidently or conda install -c conda-forge evidently.
- API and Documentation: The API reference and documentation have been enhanced for clarity, including improved logo display in API documentation and better organization of examples. The project maintains a comprehensive API reference and a growing collection of tutorials.
- The project maintains an active Discord community for discussions and support.
- Integration with Other Tools: Evidently supports integration with existing MLOps and AI observability tools, including OpenInference for tracing and GitHub Actions for CI/CD pipelines. This interoperability enhances its utility in production environments.
Evidently’s modular design allows users to start with basic evaluations and scale up to full monitoring services.