Fortuna: A Library for Uncertainty Quantification in Deep Learning
Fortuna is an open-source library developed by Amazon Web Services for uncertainty quantification in deep learning. Released in 2022 and archived in 2025, Fortuna is no longer actively maintained but remains a valuable resource for researchers and practitioners working on model calibration, conformal prediction, and Bayesian inference. The library is designed to integrate with pre-trained models across frameworks and supports Bayesian methods specifically for models built with Flax, a neural network library for JAX.
Accurate uncertainty quantification is essential in applications where model predictions directly influence critical decisions, such as healthcare diagnostics, autonomous systems, or financial forecasting. By offering calibrated uncertainty estimates, Fortuna helps users assess the reliability of predictions, trigger human intervention when necessary, or determine whether a model can be safely deployed in real-world environments.
The library operates under the Apache 2.0 license, ensuring permissive use and modification. It is implemented in Python and relies on JAX as a core dependency, which is required for installation. Fortuna’s design prioritizes usability for practitioners unfamiliar with uncertainty quantification, offering highly configurable pipelines tailored to different use cases.
How It Works
Fortuna provides three distinct usage modes, each catering to different stages of the machine learning pipeline and varying levels of compatibility requirements. These modes allow users to integrate uncertainty quantification into their workflows with minimal disruption, whether they are working with pre-trained models, raw model outputs, or custom Flax-based models.
From Uncertainty Estimates
This mode is the most accessible entry point for users who already possess uncertainty estimates from their models. It requires minimal compatibility constraints and is the fastest way to interact with Fortuna. The primary functionality in this mode involves conformal prediction methods for both classification and regression tasks. Conformal prediction adjusts uncertainty estimates to ensure that prediction sets retain a user-specified probability coverage. For example, in one-dimensional regression, conformal sets can be interpreted as calibrated confidence or credible intervals.
To use this mode, users provide lower and upper bounds of credible intervals for validation and test datasets, along with corresponding validation targets. Fortuna then generates conformal prediction intervals that align with the specified coverage error. However, if the input uncertainty estimates are inaccurate, the resulting conformal sets may be overly broad and impractical.
- Define validation and test credible intervals as arrays of lower and upper bounds.
- Provide validation targets corresponding to the validation intervals.
- Use the QuantileConformalRegressor class to generate calibrated conformal intervals for test inputs.
The code snippet below demonstrates this process:
from fortuna.conformal import QuantileConformalRegressor
conformal_intervals = QuantileConformalRegressor().conformal_interval(
vallowerbounds=vallowerbounds, valupperbounds=valupperbounds,
testlowerbounds=testlowerbounds, testupperbounds=testupperbounds,
From Model Outputs
This mode is intended for users who have trained models in any framework and possess model outputs in the form of NumPy arrays. It allows for the calibration of model outputs, uncertainty estimation, metric computation, and the generation of conformal sets. Unlike the first mode, this approach ensures that uncertainty estimates are properly calibrated, addressing potential inaccuracies in pre-existing estimates.
However, if the original model was trained using classical (non-Bayesian) methods, the resulting uncertainty quantification—particularly epistemic uncertainty—may still be inadequate. In such cases, users are advised to consider the third mode, which leverages Bayesian inference for improved uncertainty estimation.
- Provide validation and test model outputs as NumPy arrays.
- Supply corresponding validation and test target variables.
- Use the OutputCalibClassifier class to calibrate the model outputs and compute predictive entropy estimates.
The following example illustrates this process:
from fortuna.outputcalibmodel import OutputCalibClassifier
calibmodel = OutputCalibClassifier()
status = calibmodel.calibrate(outputs=valoutputs, targets=valtargets)
From Flax Models
This mode is the most advanced and requires models to be implemented using Flax, a neural network library for JAX. It enables users to replace traditional model training with scalable Bayesian inference procedures, which significantly enhance the quantification of predictive uncertainty.
- Define a Flax-based deep learning model that outputs logits.
- Prepare TensorFlow data loaders for training, validation, and calibration datasets.
- Convert these data loaders into Fortuna-compatible formats using the DataLoader class.
- Use the ProbClassifier class to train the model with Bayesian inference and generate calibrated probability estimates.
from fortuna.data import DataLoader
traindataloader = DataLoader.fromtensorflowdataloader(traindataloader)
calibdataloader = DataLoader.fromtensorflowdataloader(valdataloader)
Use Cases
Fortuna is designed to address critical challenges in uncertainty quantification across various domains where model reliability is paramount. Its three usage modes cater to different stages of the machine learning lifecycle, from post-hoc calibration to integrated Bayesian training.
Critical Decision-Making Applications
Fortuna’s calibrated uncertainty estimates enable practitioners to assess the reliability of predictions, ensuring that models are only deployed when their confidence levels meet predefined thresholds. For instance, in medical imaging, a model’s uncertainty estimate can determine whether a second opinion from a human expert is warranted.
Model Deployment and Monitoring
For production systems, Fortuna facilitates the integration of uncertainty quantification into monitoring pipelines. By continuously calibrating model outputs and generating conformal prediction sets, organizations can proactively identify degradation in model performance or shifts in data distributions.
Research and Benchmarking
Fortuna provides researchers with a standardized framework for benchmarking uncertainty quantification methods. Its support for conformal prediction and Bayesian inference allows for rigorous comparisons of different approaches, contributing to the broader understanding of how uncertainty can be effectively estimated in deep learning models. The library’s modular design also makes it suitable for educational purposes, enabling practitioners to experiment with uncertainty quantification techniques.
Integration with Amazon SageMaker
Fortuna offers seamless integration with Amazon SageMaker, a fully managed platform for building, training, and deploying machine learning models. Users can deploy Fortuna pipelines on SageMaker with minimal effort, leveraging AWS’s infrastructure for scalable training and inference.
- Building and pushing a Docker image to an Amazon ECR repository, with optional customization for additional dependencies.
- Creating an S3 bucket to store training results and model artifacts.
- Configuring a YAML file to specify AWS credentials, entrypoint scripts, arguments, and monitoring metrics.
This integration is particularly useful for teams already using SageMaker, as it streamlines the deployment of uncertainty-aware models in production environments.
Key Features and Capabilities
Fortuna’s core functionality revolves around three pillars: conformal prediction, model output calibration, and Bayesian inference for Flax models. Each of these features addresses specific challenges in uncertainty quantification, providing users with flexible tools tailored to their workflows.
Conformal Prediction Methods
Conformal prediction is a non-parametric approach that ensures prediction sets achieve a user-specified coverage probability. Fortuna implements conformal methods for both classification and regression tasks, making it possible to generate rigorous uncertainty estimates without requiring access to the model’s internal parameters. This feature is particularly useful for users who need to work with pre-trained models from third-party frameworks, as it does not impose additional training requirements.
In regression tasks, conformal prediction intervals can be interpreted as calibrated versions of traditional confidence or credible intervals. This property is critical for applications where overconfidence in predictions could lead to costly errors.
Model Output Calibration
By recalibrating these outputs, Fortuna ensures that predicted probabilities accurately reflect the model’s true uncertainty.
This approach is computationally efficient and does not require retraining the original model. Users can then compute metrics such as predictive entropy, which quantifies the average uncertainty of the model’s predictions. This feature is especially valuable for practitioners who need to deploy models quickly but still require reliable uncertainty estimates.
Bayesian Inference for Flax Models
For users working with Flax-based models, Fortuna provides tools to perform scalable Bayesian inference. Unlike classical training methods, Bayesian approaches explicitly model uncertainty over model parameters, leading to more accurate estimates of both aleatoric (data-driven) and epistemic (model-related) uncertainty.
By replacing standard training with Bayesian inference, users can obtain well-calibrated probability estimates that better reflect the true uncertainty in their predictions. However, this approach requires models to be implemented in Flax, which may limit its applicability for users working with other frameworks.
Highly Configurable and Intuitive Design
Fortuna’s API is designed to be intuitive for practitioners with varying levels of expertise in uncertainty quantification. The library provides clear documentation, including quickstart guides, examples, and references to key classes and methods. Users can easily configure pipelines to suit their specific needs, whether they are working with pre-trained models, raw outputs, or custom Flax implementations.
The library’s modular structure allows for incremental adoption. Users can start with the simplest mode (From Uncertainty Estimates) and gradually incorporate more advanced techniques as their requirements evolve. This flexibility makes Fortuna accessible to both researchers exploring new methods and industry practitioners looking to deploy uncertainty-aware models in production.
Integration with Popular Tools and Platforms
Fortuna is compatible with a range of tools and platforms commonly used in the machine learning ecosystem. Additionally, Fortuna supports integration with Hugging Face’s Transformers library, enabling users to apply uncertainty quantification techniques to pre-trained language models.
For deployment, Fortuna offers native support for Amazon SageMaker, allowing users to run uncertainty quantification pipelines on AWS’s managed infrastructure.
Installation and Dependencies
Fortuna can be installed via pip or Poetry, with JAX as a mandatory dependency. The library also supports optional dependencies for additional functionality, such as:
- Transformers for working with Hugging Face models.
- SageMaker for deploying Fortuna pipelines on AWS.
- Sphinx for building documentation.
The installation process via pip is straightforward: users simply execute pip install aws-fortuna. For more complex setups, Poetry can be used to manage dependencies and virtual environments. Poetry supports additional flags to customize the installation, such as -E transformers for Hugging Face compatibility or -E sagemaker for AWS integration.
Users are encouraged to review the official documentation for detailed installation instructions, including troubleshooting tips for common issues. The documentation also provides guidance on setting up Fortuna in different environments, such as local development machines or cloud-based platforms like SageMaker.