Quantus: A Toolkit for Evaluating Neural Network Explanations
Quantus is a Python-based toolkit developed by the Understandable Machine Intelligence Lab at TU Berlin, designed to quantitatively evaluate the quality of explanations generated by neural networks. Published in the Journal of Machine Learning Research (JMLR) in 2023, Quantus addresses a critical gap in explainable AI (XAI) research by providing a structured framework for assessing explanations beyond qualitative comparisons. The toolkit supports frameworks like PyTorch and TensorFlow and is actively maintained, with recent updates improving performance, scalability, and compatibility.
The core motivation behind Quantus stems from the limitations of visual comparisons in XAI evaluation. While methods such as Saliency, Integrated Gradients, GradientSHAP, and FusionGrad can be qualitatively compared, these comparisons often lack depth. Quantus introduces a systematic approach by quantifying explanations across multiple criteria, including faithfulness, robustness, localization, complexity, randomisation, and axiomatic properties. This allows researchers to move beyond superficial assessments and gain insights into how different parameters—such as pixel replacement strategies in faithfulness tests—affect the ranking and reliability of explanation methods.
The toolkit is built on the principle that explanations should be evaluated rigorously, especially in the absence of ground truth data. By automating the quantification of XAI metrics, Quantus helps standardize evaluation practices, ensuring that explanations are not only interpretable but also trustworthy and aligned with model behavior.
How It Works
These categories include faithfulness, robustness, localization, complexity, randomisation, and axiomatic metrics. Each metric is implemented to measure a particular property of explanations, such as how well they align with model predictions (faithfulness) or how stable they remain under input perturbations (robustness).
The toolkit is designed to be modular, allowing users to apply individual metrics or combine them for comprehensive evaluation.
For instance, metrics such as PixelFlipping, Monotonicity, and FaithfulnessCorrelation now support batched evaluations, delivering a 12-fold speedup in runtime efficiency.
Users are encouraged to participate in its improvement through the project’s Discord channel or by submitting pull requests. Documentation and tutorials are provided to guide users through installation, usage, and customization, ensuring accessibility for both beginners and experienced practitioners.
Use Cases
Quantus is primarily used in research and development environments where the evaluation of neural network explanations is critical.
Metrics like Sparseness and Complexity help determine whether explanations are concise and focused on the most predictive features, reducing the risk of overfitting or spurious correlations.
Beyond technical applications, Quantus supports responsible AI practices by providing a standardized framework for evaluating explanations. By offering a comprehensive suite of metrics, Quantus helps researchers and practitioners move toward more accountable and interpretable AI systems.
Key Features and Capabilities
Quantus includes over 35 metrics distributed across six evaluation categories, each designed to address a specific aspect of explanation quality.
- Faithfulness Metrics: These measure how closely explanations align with the model’s predictive behavior. Examples include Faithfulness Correlation, which assesses the correlation between attribution scores and changes in model output, and Pixel Flipping, which evaluates the impact of perturbing pixels on classification scores.
- Robustness Metrics: These evaluate the stability of explanations under input perturbations.
- Localization Metrics: These assess whether explanations correctly identify regions of interest, such as objects in images or relevant features in tabular data.
- Complexity Metrics: These capture the conciseness of explanations, ensuring that only the most predictive features are highlighted. Other metrics in this category include Complexity and Entropy.
- Randomisation Metrics: These evaluate the sensitivity of explanations to random perturbations, ensuring that they are not overly dependent on noise or spurious patterns.
- Axiomatic Metrics: These evaluate explanations based on formal properties, such as completeness, sensitivity, and implementation invariance. While fewer in number, these metrics provide a theoretical foundation for evaluating explanations beyond empirical observations.
The toolkit’s modular design enables users to select specific metrics based on their use case, whether they require a broad overview or a focused assessment of particular explanation properties.
This feature accelerates computations by processing multiple samples simultaneously, reducing runtime by up to 12 times for metrics like PixelFlipping and FaithfulnessCorrelation.
The toolkit is licensed under a custom or multiple licenses, depending on the specific use case, and is available on GitHub. Users are encouraged to cite the original paper and relevant metric authors when using Quantus in their research. The project’s documentation provides detailed guides on installation, usage, and contribution, ensuring that users can integrate Quantus seamlessly into their workflows.