How It Works
NannyML is an open-source Python library designed to address critical challenges in post-deployment machine learning monitoring. It operates by estimating model performance without requiring ground truth labels, a capability achieved through two core algorithms: Confidence-Based Performance Estimation (CBPE) for classification tasks and Direct Loss Estimation (DLE) for regression tasks. These algorithms evaluate the expected performance of predictions made at inference time, providing metrics such as ROC AUC for classification or RMSE for regression. Unlike traditional methods, CBPE and DLE do not rely on future predictions but instead assess the performance of predictions already generated during deployment.
For data drift detection, NannyML employs a PCA-based data reconstruction approach to identify multivariate feature drift. This method monitors changes in reconstruction error over time, triggering alerts when the error exceeds a threshold calculated from a reference period. Additionally, NannyML utilizes statistical tests, including Jensen-Shannon Distance and L-Infinity Distance, to detect univariate feature drift and model output drift. These tests are applied to temporal feature distributions, with results corrected for multiplicity to avoid false positives. The library also supports monitoring target distribution drift, though this requires the presence of actual target labels.
A key innovation in NannyML is its ability to link data drift alerts to performance changes, reducing alert fatigue by filtering out drift notifications that do not impact model performance. This is achieved through an intelligent alerting system that prioritizes alerts based on their relevance to performance degradation. The library further supports custom prioritization criteria via a Ranker module, allowing users to define alert thresholds and ranking logic tailored to their specific needs.
NannyML’s architecture is model-agnostic, meaning it can be applied to any tabular data use case, including both classification and regression problems. The library is built to integrate seamlessly into existing workflows, offering interactive visualizations and an intuitive interface for data scientists. Its design ensures compatibility with various environments, from local development to cloud-based deployments, with support for database connections and Docker-based installations.
This modularity allows for flexible deployment and customization.
NannyML’s algorithms are grounded in research-driven methodologies, ensuring accuracy and reliability in performance estimation and drift detection. The PCA-based drift detection method, for instance, provides a robust way to identify structural changes in multivariate data, while the statistical tests for univariate drift offer granular insights into feature-level shifts. By combining these approaches, NannyML enables data scientists to maintain visibility into model behavior, even in the absence of ground truth, and to take proactive measures when performance degradation is detected.
Recent updates, such as those in version 0.13.1, have focused on dependency management and compatibility, including support for Python 3.12 and fixes for issues like incorrect default thresholds in drift calculators.
Use Cases
NannyML is primarily used in scenarios where traditional model monitoring fails due to the absence or delay of ground truth labels. One common application is in production environments where target labels are unavailable or arrive with significant latency, such as in fraud detection systems, recommendation engines, or predictive maintenance models. In these cases, NannyML’s performance estimation algorithms provide a way to assess model accuracy without waiting for labeled data, enabling real-time or near-real-time monitoring.
Another key use case is detecting silent model failures, where a model’s performance degrades over time due to data drift or concept drift, but the degradation is not immediately apparent. NannyML’s drift detection and performance estimation capabilities allow teams to identify these issues early, before they result in business impact. By linking drift alerts to performance changes, NannyML helps prioritize investigations and reduces the risk of false alarms.
NannyML’s ability to estimate performance without ground truth labels supports compliance with regulations like GDPR or HIPAA, where access to sensitive target data may be restricted.
In industries where business impact is directly tied to model performance, such as finance or supply chain management, NannyML provides actionable insights into how data shifts affect operational outcomes. NannyML’s performance-centric workflow allows teams to quantify the impact of drift on model predictions and to trigger retraining or intervention when performance thresholds are breached. This proactive approach minimizes downtime and ensures that models remain aligned with business objectives.
The library is also used in research and experimental settings, where ground truth labels may be scarce or difficult to obtain. For instance, in A/B testing or online learning scenarios, NannyML can estimate the performance of models in real-time, providing feedback for iterative improvements. Its support for both classification and regression tasks makes it versatile for a wide range of experimental designs, while its open-source nature encourages collaboration and customization within research communities.
Key Features & Capabilities
- Performance Estimation Without Ground Truth: NannyML’s CBPE (Confidence-Based Performance Estimation) and DLE (Direct Loss Estimation) algorithms estimate model performance metrics such as ROC AUC or RMSE without requiring access to target labels. This is particularly useful in scenarios where labels are delayed, unavailable, or sensitive. The algorithms work by analyzing the confidence or loss associated with predictions made at inference time, providing a proxy for true model accuracy.
- Multivariate and Univariate Drift Detection: The library detects data drift using a combination of PCA-based reconstruction error for multivariate drift and statistical tests (e.g., Jensen-Shannon Distance, L-Infinity Distance) for univariate drift. Multivariate drift detection identifies structural changes in the data distribution, while univariate tests pinpoint specific features contributing to drift. This dual approach ensures comprehensive coverage of potential data shifts, from broad structural changes to granular feature-level variations.
- Intelligent Alerting and Prioritization: NannyML’s alerting system is designed to minimize false positives by linking drift alerts to actual performance degradation. By filtering out irrelevant alerts, NannyML reduces alert fatigue and ensures that teams focus only on actionable issues.
- Model-Agnostic and Scalable Design: NannyML is compatible with any tabular data model, regardless of the underlying algorithm (e.g., decision trees, neural networks, or ensemble methods).
- Interactive Visualizations and Reporting: NannyML provides built-in visualization tools to display performance estimates, drift trends, and alert rankings. These visualizations help data scientists quickly identify patterns, such as correlations between specific features and performance drops, or temporal trends in drift magnitude. The library’s documentation includes examples of how to generate and interpret these visualizations, facilitating adoption and integration into existing workflows.
- Integration with Databases and Cloud Environments: NannyML supports database connections for reading model inputs and outputs, as well as exporting monitoring results to databases. This capability is particularly useful for teams managing large-scale deployments or operating in cloud-based infrastructures. The library can be installed via PyPI, Conda, or Docker, with additional dependencies available for database support.
- Continuous Development and Community Support: NannyML is actively maintained by a team of researchers and contributors who regularly update its algorithms, fix bugs, and expand its feature set. Recent releases, such as version 0.13.1, have focused on improving dependency management, adding support for newer Python versions (e.g., Python 3.12), and addressing issues like incorrect threshold defaults. The project also maintains a Slack community and a GitHub repository for collaboration, documentation, and contributions.
- Research-Backed Algorithms: The core algorithms in NannyML, including CBPE, DLE, and PCA-based drift detection, are based on peer-reviewed research. For example, the PCA-based approach to multivariate drift detection has been shown to effectively capture complex data shifts that univariate methods might miss.
- Support for Realized Performance Tracking: While NannyML excels in estimating performance without ground truth, it also supports tracking realized performance once target labels become available. For instance, a model might initially rely on CBPE for performance estimation but later switch to realized metrics once labels are obtained.
- Customizable Thresholds and Chunking: Users can define custom thresholds for drift detection and performance alerts, tailoring the sensitivity of the monitoring system to their specific needs.
- Comprehensive Documentation and Examples: NannyML’s documentation includes detailed guides on installation, usage, and advanced features, as well as real-world examples such as the California Housing dataset and NYC Green Taxi dataset. The documentation also covers key concepts, technical references, and best practices for post-deployment monitoring.
- Compatibility with NannyML Cloud: While NannyML is an open-source library, it integrates with NannyML Cloud, a commercial offering that extends its capabilities with additional features such as automated infrastructure for alerting, scheduling, and data collection. NannyML Cloud provides a performance-centric workflow, focusing on business impact and concept drift detection, while maintaining the same core algorithms and methodologies as the open-source version. This dual offering ensures that users can choose between self-hosted and cloud-based solutions based on their needs.