Deepchecks: Continuous Validation for AI and ML
Deepchecks is an open-source Python package designed for continuous validation of machine learning models and datasets. It provides a comprehensive framework for testing data integrity, detecting train-test drift, evaluating model performance, and conducting specialized checks for natural language processing (NLP) tasks. The package supports integration into notebook environments or continuous integration (CI) pipelines, ensuring validation is embedded throughout the ML lifecycle. A commercial platform extension offers additional capabilities, including large language model (LLM) evaluation and production monitoring.
Deepchecks is actively maintained with contributions from a community of developers, researchers, and practitioners.
Each component is designed to address specific validation needs at different stages of the ML workflow, from research to deployment.
How It Works
Deepchecks operates through a system of pre-built and customizable Checks, which are modular tests for identifying issues in datasets and model performance. These Checks are categorized based on the type of data or model being validated, including tabular data, NLP, and computer vision (CV). Users can run Checks individually or group them into Suites for comprehensive validation.
Checks are implemented to detect a wide range of potential problems, such as data distribution shifts, label conflicts, weak model segments, and performance degradation over time. Each Check includes configurable parameters, allowing users to tailor tests to their specific use cases. Custom Checks can also be developed to address unique validation requirements.
The results of Checks can be consumed in multiple formats: visual reports (HTML or Jupyter notebooks), programmatic outputs (Python or JSON), or through Deepchecks’ dynamic user interface (UI).
For production environments, Deepchecks Monitoring provides real-time tracking of model behavior, enabling continuous validation of deployed models. Alerts can be configured based on Check results, ensuring proactive identification of anomalies or performance degradation. The monitoring component supports both open-source and commercial deployments, with the latter offering advanced features such as LLM-specific evaluations.
Use Cases
- Data Integrity Validation: Ensuring datasets are clean, consistent, and free of anomalies or conflicts.
- Train-Test Drift Detection: Monitoring changes in data distribution between training and test sets, or between historical and production data. This is critical for maintaining model reliability in dynamic environments.
- Model Evaluation: Assessing model performance across different segments of the data, such as identifying subgroups where the model underperforms.
- NLP-Specific Validation: Specialized Checks for text data, such as detecting biases, evaluating sentiment consistency, or assessing the impact of preprocessing steps on model performance.
- CI/CD Integration: Automating validation steps within CI pipelines to ensure models meet quality standards before deployment. This reduces the risk of deploying flawed models into production.
- Production Monitoring: Continuously validating deployed models to detect performance degradation, data drift, or concept drift in real-time. Alerts can trigger automated responses, such as retraining or model updates.
The package’s support for notebook environments (e.g., Jupyter) and CI systems (e.g., GitHub Actions) makes it accessible for both research and production teams. For example, a data scientist can run a Suite of Checks on a dataset in a notebook to identify issues before training begins, while a DevOps team can integrate Deepchecks into their CI pipeline to enforce validation gates.
Key Features and Capabilities
Deepchecks provides a structured approach to ML validation through its Checks, Suites, and monitoring tools. Key features include:
- Pre-Built Checks: Over 100 pre-implemented Checks covering data quality, model performance, and domain-specific issues (e.g., NLP or CV).
- Customizable Checks: Users can extend or modify existing Checks to fit specific requirements.
- Suites for Organized Validation: Checks can be grouped into Suites, which run sequentially and produce consolidated reports. Suites are available for common use cases, such as model evaluation or data integrity, and can be combined or customized as needed.
- Multi-Format Outputs: Results are accessible via HTML reports, Jupyter widgets, or programmatic outputs (Python objects or JSON).
- Production Monitoring: The monitoring component tracks model performance and data drift in real-time. It supports alerting mechanisms and a web-based UI for visualizing validation results over time.
- LLM Evaluation (Commercial): The commercial platform extends Deepchecks’ capabilities to include specialized evaluations for large language models, such as assessing hallucination rates, bias, or coherence in generated text.
- Open-Source and Commercial Offerings: While the core Testing and Monitoring components are open-source under AGPL 3.0, certain advanced features (e.g., LLM monitoring) require a commercial license. An open-source-only version of the monitoring component is also available for users who prefer to avoid proprietary dependencies.
The package is designed to be extensible, with clear documentation and contribution guidelines. Users can submit bug reports, feature requests, or pull requests to improve the tool.
Installation is straightforward, with support for Python’s pip and conda package managers. Additional submodules are available for NLP (deepchecks[nlp]) and computer vision (deepchecks[vision]). Monitoring can be deployed locally via Docker or as a managed SaaS service.
Installation and Setup
Deepchecks Testing can be installed via pip with the command pip install deepchecks -U --user. For specialized use cases, additional submodules are required:
For production monitoring, users can deploy the open-source version locally using Docker. The installation process involves running pip install deepchecks-installer followed by deepchecks-installer install-monitoring, which automates dependency installation and starts the application. The commercial version offers additional features and is accessible via a SaaS platform or self-hosted deployment with a license.
Quickstart guides are provided for tabular, NLP, and vision data types, demonstrating how to create dataset objects (e.g., Dataset, TextData, VisionData) and run Checks or Suites.
from deepchecks.tabular.suites import modelevaluation
suite = modelevaluation()
suiteresult = suite.run(traindataset=traindataset, testdataset=testdataset, model=model)
This snippet runs a model evaluation Suite and saves the results as an HTML report. Users can also visualize results inline in Jupyter or process them programmatically.
Community and Contributions
Deepchecks is maintained by a collaborative community of contributors, including developers, researchers, and ML practitioners. The project encourages contributions in the form of bug fixes, feature enhancements, documentation improvements, or community engagement.
Notable contributors include Itay Gabbay, Noam Bressler, JKL98ISR, and others, who have worked on core functionality, documentation, and testing. The project also maintains a Slack workspace for real-time discussions, support, and networking among users and maintainers.