Data Logging Library for Statistical Profiling and Monitoring
whylogs is an open-source Python library designed to create lightweight statistical profiles of datasets and model inputs and outputs. Unlike traditional data logging methods, whylogs avoids storing raw data while enabling drift detection, data quality validation, and monitoring. Developed by WhyLabs since 2020 under the Apache 2.0 license, it serves as a foundational tool for monitoring and observability in machine learning workflows.
The library is positioned as an open standard for data logging, offering a structured approach to capturing key statistical properties of datasets. These properties include distributions, missing values, and customizable metrics, allowing users to generate profiles that serve as efficient snapshots of their data. This approach is particularly valuable in environments where data privacy, scalability, and real-time monitoring are critical.
Its design prioritizes efficiency, customization, and mergeability, making it suitable for both small-scale experiments and large-scale distributed systems.
How It Works
At the core of whylogs is the concept of a whylogs profile, which captures statistical properties of datasets without storing the raw data. These profiles are generated by logging datasets through the library, which computes summary statistics such as distributions, missing values, and custom metrics. The profiles are designed to be compact yet highly representative of the original data, offering a balance between fidelity and efficiency.
Profiles are generated using a simple Python API. For example, logging a Pandas DataFrame involves importing the library and executing a single function call: why.log(df). This generates a profile that can be further analyzed, visualized, or sent to external platforms like WhyLabs for automated monitoring. The process is supported in both Python and Java environments, ensuring broad compatibility across different workflows.
Benchmarks indicate that profiling time per instance can range from 33 seconds to 2.7 minutes, depending on the scale and parallelization. For large datasets with thousands of features and high query volumes (10M+ QPS), the overhead typically remains under 5%, with distributed environments like Ray or Apache Spark further optimizing performance through parallelization.
Profiles are also mergeable, allowing users to combine multiple profiles into a single aggregated profile. This feature is particularly useful in distributed and streaming systems, where data is processed across multiple nodes or over time.
Use Cases
whylogs is employed across a variety of use cases in machine learning and data engineering, primarily focused on ensuring data reliability and model performance.
- Drift Detection: Monitoring changes in data distributions over time to identify training-serving skew, concept drift, or model performance degradation. This is achieved by comparing profiles of model inputs between training and production environments.
- Data Quality Validation: Setting constraints on datasets to ensure they meet expected criteria, such as numerical ranges or text formats.
- Exploratory Data Analysis (EDA): Generating interactive visualizations of dataset statistics directly in Jupyter notebooks. This allows users to explore distributions, missing values, and other metrics without handling raw data.
- Model Observability: Tracking data distributions and quality metrics for machine learning experiments. Profiles provide a baseline for evaluating model inputs and outputs, ensuring consistency and reliability.
- Profiles serve as a reproducible record of dataset characteristics, facilitating compliance and transparency.
- Performance Monitoring: Integrating with WhyLabs to set up automated alerts for data drift or quality issues. The platform ingests whylogs profiles and provides observability into data and ML systems, reducing manual oversight.
The library’s support for both structured and unstructured data—including tabular, image, text, and embeddings—expands its applicability to diverse workflows. For instance, image data can be logged using custom trackers, while text data can be analyzed for string features or JSON formats.
Key Features and Capabilities
whylogs profiles are built on three core properties: efficiency, customization, and mergeability. These properties address common challenges in data logging and monitoring, such as scalability, flexibility, and integration with existing systems.
Efficiency is achieved through the use of highly scalable streaming algorithms that compute statistics in-line with data processing. This minimizes compute overhead, with most use cases experiencing less than 1% overhead.
Customization is a defining feature of whylogs, allowing users to define specific metrics and constraints tailored to their data types and use cases. For example, users can track custom performance metrics or enforce constraints on numerical ranges, text formats, or categorical distributions. This flexibility extends to complex data types like images and embeddings, where domain-specific trackers can be implemented.
Mergeability enables whylogs to function effectively in distributed systems. Profiles can be combined across nodes or time periods, providing aggregated views of data without requiring centralized storage. This is particularly valuable in streaming pipelines, where data is continuously generated and processed.
Beyond profiling, whylogs offers tools for data validation and visualization. Data constraints allow users to define rules for their datasets, such as ensuring numerical values fall within specified ranges or that text adheres to expected formats. Violations can trigger alerts or fail unit tests, integrating seamlessly with CI/CD pipelines. The Profile Visualizer module enables interactive exploration of profiles in Jupyter notebooks, supporting drift analysis, exploratory data analysis, and model debugging.
Integration with WhyLabs extends whylogs’ functionality by providing a managed service for automated monitoring. Users can send profiles to WhyLabs to set up alerts for data drift, quality issues, or performance degradation. The platform offers a free Starter edition, allowing users to experience its full capabilities without financial commitment. This integration bridges local profiling with cloud-based observability, creating a unified approach to data and model monitoring.
whylogs also supports a range of data types, including tabular, image, text, and embeddings. For instance, image data can be logged using custom trackers that capture features like pixel distributions or embeddings, while text data can be analyzed for string properties or JSON structures.
The library’s design emphasizes minimalism and scalability, with profiles generated directly in data pipelines or sidecar containers. This approach reduces latency and ensures that logging does not disrupt production workflows. Additionally, whylogs collects anonymous usage statistics by default, which can be disabled by setting the WHYLOGS_NO_ANALYTICS environment variable to True. This balances the need for insights into library adoption with user privacy.
Updates include fixes for API versioning, token handling, and performance optimizations, ensuring that whylogs remains reliable and up-to-date with evolving data logging requirements.