Introduction
crepes is a Python package designed for conformal prediction, a statistical framework that transforms predictions from machine learning models into sets, intervals, or distributions with rigorous coverage guarantees. Developed by Henrik Boström at KTH, the package supports conformal classifiers, regressors, and predictive systems, ensuring that predictions adhere to specified confidence levels regardless of the underlying model. Released under the permissive BSD 3-Clause license, crepes integrates seamlessly with scikit-learn compatible models, providing tools for uncertainty quantification and model calibration.
The package implements standard and advanced variants of conformal prediction, including Mondrian and normalized methods, which allow for group-specific calibration and adaptive uncertainty estimation. It also includes utilities for testing exchangeability assumptions and semi-online calibration, making it versatile for both static and dynamic prediction scenarios. With over 580 stars on GitHub, crepes has gained recognition for its contributions to interpretable and explainable AI.
How It Works
crepes operates by wrapping existing classifiers or regressors and augmenting their predictions with conformal inference. The process begins with a standard training set, which is further divided into a proper training set and a calibration set. The model is trained on the proper training set, and the calibration set is used to compute non-conformity scores or difficulty estimates, which are then used to generate well-calibrated prediction sets, intervals, or p-values.
For classification tasks, crepes produces prediction sets that include all plausible classes with guaranteed coverage. For regression, it generates prediction intervals that account for uncertainty, optionally normalized by difficulty estimates to reflect varying levels of confidence across instances. The package supports both inductive and semi-online conformal predictors, where the latter updates the calibration set iteratively to maintain independence of predictions.
The crepes.extras module provides predefined difficulty estimators, non-conformity score functions, and Mondrian categorizers, while the crepes.martingales module offers tools for testing exchangeability assumptions and semi-online p-value generation.
Use Cases
For example, in medical decision-making, conformal prediction sets can provide actionable uncertainty estimates for diagnostic models, ensuring that predictions are reliable at specified confidence levels. Similarly, in finance, prediction intervals for asset prices or risk metrics can be adjusted dynamically to reflect changing market conditions.
The package’s support for Mondrian conformal classifiers enables subgroup-specific calibration, which is valuable in scenarios where data exhibits heterogeneous uncertainty across different groups.
Semi-online conformal predictors are suited for real-time applications where data arrives sequentially, such as streaming analytics or online recommendation systems. By updating the calibration set after each prediction, these methods ensure that subsequent predictions remain statistically valid, even as the underlying data distribution evolves.
Key Features & Capabilities
- Standard and Advanced Conformal Prediction: Implements standard, Mondrian, and normalized conformal classifiers and regressors, allowing for both global and subgroup-specific calibration. Mondrian methods use user-defined or predefined categorizers to tailor prediction sets to specific groups, while normalized regressors adjust interval widths based on difficulty estimates.
- Flexible Non-Conformity Scoring: Users can define custom non-conformity scores or difficulty estimates, or utilize predefined functions from crepes.extras. For example, difficulty can be estimated using nearest-neighbor standard deviations or other domain-specific metrics, enabling fine-grained control over uncertainty quantification.
- Semi-Online Calibration: Supports iterative updates to the calibration set, making predictions statistically independent over time. This is achieved through methods like predict_p_online and predict_int_online, which are essential for dynamic environments where data distributions shift.
- The evaluate method computes these metrics for both offline and online calibration scenarios.
- Integration with scikit-learn: Designed to work with any scikit-learn compatible model, including RandomForest, SVM, or neural networks. The WrapClassifier and WrapRegressor classes serve as drop-in replacements, requiring minimal additional code to enable conformal inference.
- Exchangeability Testing: The crepes.martingales module includes tools like SimpleJumper, SleeperStayer, and CompositeMartingale to test the exchangeability assumption, which is foundational to conformal prediction.
- Users can also obtain smoothed or non-smoothed p-values, which are useful for hypothesis testing or model comparison.
- The CompositeMartingale class has also been optimized to terminate early when thresholds are reached, reducing unnecessary computations.
- Documentation and Examples: Comprehensive documentation and Jupyter notebooks, such as martingales_nb.ipynb, provide practical guidance on implementing conformal prediction. Quickstart examples demonstrate usage with datasets from OpenML, covering classification (e.g., QSAR Biodeg) and regression (e.g., House Sales) tasks.