Introduction
MAPIE is an open-source Python library designed for quantifying uncertainties and controlling the risks associated with machine learning models. Developed as part of the scikit-learn-contrib ecosystem, MAPIE provides tools for conformal prediction, enabling the generation of prediction intervals and sets with guaranteed coverage for regression, classification, and time series tasks. The library also supports calibration, ensuring that probabilistic predictions align with observed frequencies. MAPIE was first released in 2021 and has since evolved to include advanced features such as risk control for complex tasks like multi-label classification and semantic segmentation, as well as exchangeability tests to validate the applicability of conformal prediction methods.
The library is built on principles from conformal prediction and distribution-free inference, implementing peer-reviewed algorithms that offer theoretical guarantees under minimal assumptions about data and models. This makes MAPIE versatile for integration with various machine learning frameworks, including scikit-learn, TensorFlow, and PyTorch, through scikit-learn-compatible wrappers.
MAPIE is licensed under the permissive BSD 3-Clause license, ensuring accessibility for both research and industry applications. The library’s documentation and examples have been continuously updated, with significant API changes introduced in version 1.0 (2025) and ongoing enhancements in subsequent releases, such as version 1.5.0 (2026), which added conditional conformal prediction and adaptive risk control methods.
How It Works
MAPIE operates by estimating model uncertainty on a conformalization dataset, which is separate from the training data. This process allows the library to compute prediction intervals for regression tasks, prediction sets for classification tasks, and probabilistic guarantees for metrics like recall and precision in complex scenarios. The core mechanism relies on conformal prediction, a distribution-free approach that ensures coverage guarantees without relying on strong assumptions about the underlying data distribution.
For regression and time series tasks, MAPIE generates prediction intervals that account for uncertainty in model predictions. In classification, it produces prediction sets that include the true class with a specified confidence level. The library supports both single-label and multi-label classification, as well as semantic segmentation in computer vision, where it provides probabilistic guarantees for precision and recall. MAPIE’s compatibility with scikit-learn ensures seamless integration with existing workflows, while its theoretical foundation in conformal prediction and distribution-free inference guarantees reliability across diverse use cases.
Key components of MAPIE include algorithms for cross-conformal prediction, jackknife-after-bootstrap, and adaptive conformal risk control. The library also introduces exchangeability tests, such as permutation and online martingale tests, to validate whether conformal prediction can be legitimately applied to a given dataset. This ensures that users can confidently deploy MAPIE in scenarios where its assumptions hold.
MAPIE’s architecture is modular, allowing users to customize uncertainty quantification and risk control for specific applications. For example, the library supports conditional conformal prediction, which adapts prediction intervals or sets based on additional contextual features. This is particularly useful in dynamic environments, such as time-series forecasting or adaptive decision-making systems. The library’s documentation provides detailed examples and theoretical explanations, making it accessible to both researchers and practitioners.
Use Cases
MAPIE is applied across a range of domains where uncertainty quantification and risk control are critical. In regression tasks, such as financial forecasting or demand prediction, MAPIE generates prediction intervals that account for model uncertainty, enabling more robust decision-making under uncertainty. For classification tasks, including medical diagnosis or fraud detection, the library produces prediction sets that include the true class with a specified confidence level, reducing the risk of incorrect predictions.
One of the library’s strengths is its ability to handle complex tasks like multi-label classification and semantic segmentation. In multi-label classification, where each instance can belong to multiple classes, MAPIE provides probabilistic guarantees for metrics like precision and recall, ensuring reliable performance in applications such as image tagging or document classification. In semantic segmentation, where pixel-level predictions are required, MAPIE extends conformal prediction to provide uncertainty estimates for each segment, improving the reliability of computer vision systems.
Emerging use cases for MAPIE include large language models (LLMs) as judges and adaptive risk control in dynamic environments. Similarly, in time-series forecasting or real-time decision-making systems, the library’s adaptive conformal prediction methods allow for continuous monitoring and adjustment of prediction intervals based on evolving data distributions.
The library’s scikit-learn-compatible wrappers simplify integration, allowing users to apply conformal prediction to pre-trained models without extensive modifications. This flexibility makes MAPIE a valuable tool for researchers and practitioners working on cutting-edge machine learning applications where uncertainty and risk management are paramount.
Key Features & Capabilities
- Prediction Intervals and Sets: MAPIE computes prediction intervals for regression and time series tasks, as well as prediction sets for classification tasks, ensuring that the true outcome is included within the predicted interval or set with a specified confidence level. This is achieved through conformal prediction methods like cross-conformal prediction and jackknife-after-bootstrap, which are distribution-free and model-agnostic.
- Risk Control for Complex Tasks: The library extends conformal prediction to complex scenarios, including multi-label classification and semantic segmentation. In these cases, MAPIE provides probabilistic guarantees for metrics such as precision and recall, enabling reliable performance in applications where traditional uncertainty quantification methods fall short. For instance, in multi-label classification, MAPIE ensures that the predicted set of labels includes the true labels with high probability, reducing the risk of false positives or negatives.
- Exchangeability Testing: MAPIE includes tools for testing exchangeability assumptions, which are critical for the validity of conformal prediction. This feature helps users avoid misapplying MAPIE in scenarios where its guarantees do not hold, ensuring more reliable and interpretable results.
- Conditional and Adaptive Methods: Recent updates to MAPIE introduce conditional conformal prediction and adaptive risk control methods. Conditional prediction intervals or sets are computed based on additional features, allowing for more nuanced uncertainty quantification in dynamic environments. Adaptive conformal risk control adjusts prediction intervals in real-time, making MAPIE suitable for applications like time-series forecasting or online decision-making systems where data distributions evolve over time.
- Integration with Machine Learning Frameworks: MAPIE is designed to work seamlessly with popular machine learning libraries, including scikit-learn, TensorFlow, and PyTorch. Its scikit-learn-compatible wrappers allow users to apply conformal prediction to any model within these ecosystems, without requiring modifications to the underlying architecture. This compatibility extends MAPIE’s utility to a wide range of applications, from traditional statistical models to deep neural networks.
- These metrics help users assess the performance of their models in terms of uncertainty estimation and risk control.
- Documentation and Examples: MAPIE’s documentation is comprehensive, featuring hands-on examples for regression, classification, time series, and conditional prediction tasks. Users can explore guided exercises, such as regression and LLM conformal prediction, to understand how to apply MAPIE in their own projects. The library’s examples cover a wide range of scenarios, from basic uncertainty quantification to advanced risk control in complex tasks.
- Version Updates and Community Contributions: MAPIE has undergone significant updates since its initial release, with version 1.0 introducing major API changes and version 1.5.0 adding features like conditional conformal prediction and adaptive risk control. The library’s development is community-driven, with contributions from researchers and practitioners worldwide. Users are encouraged to participate in the project by opening issues, contributing code, or proposing new features, ensuring that MAPIE remains at the forefront of uncertainty quantification and risk control in machine learning.