Model Interpretability Library for PyTorch
Captum is an open-source library developed by Meta, designed to provide interpretability and explainability for PyTorch models. Released in 2019 under the permissive BSD 3-Clause license, Captum supports a wide range of attribution algorithms across modalities such as vision, text, and other domains. The library is structured to integrate seamlessly with existing PyTorch models, requiring minimal modifications to the original neural network architecture. Its primary purpose is to facilitate interpretability research by offering a generic framework for implementing and benchmarking new algorithms.
The library’s design emphasizes accessibility, allowing researchers and practitioners to easily adopt and extend its capabilities. Captum’s modular architecture enables the integration of new interpretability methods, fostering collaboration within the research community.
Captum’s compatibility extends to most PyTorch model types, ensuring broad applicability across different use cases. Whether working with convolutional neural networks for image analysis, transformers for natural language processing, or other architectures, Captum provides the necessary tools to dissect model behavior and understand feature contributions.
How It Works
Captum organizes its attribution algorithms into three primary categories: primary attribution, layer attribution, and neuron attribution. Each category addresses a specific aspect of model interpretability, providing granular insights into how inputs influence model outputs.
Primary attribution evaluates the contribution of each input feature to the final output of the model. This category includes methods such as Integrated Gradients, Gradient SHAP, and Saliency, which quantify the impact of individual features on predictions. For example, Integrated Gradients approximates the integral of gradients along a path from a baseline input to the actual input, adhering to axioms like sensitivity and implementation invariance.
Layer attribution focuses on assessing the contribution of neurons within a specific layer to the model’s output. Techniques like Guided Backpropagation and Deconvolution are tailored for convolutional networks, where spatial relationships between features are critical. These methods modify the backpropagation process to highlight relevant regions in the input space, such as identifying which pixels in an image most influence a classification decision. Guided GradCAM, for instance, combines the high-resolution insights of Guided Backpropagation with the class-discriminative strengths of GradCAM, offering a balanced approach for convolutional models.
Neuron attribution delves deeper into the internal mechanics of the model by evaluating how input features affect the activation of specific hidden neurons. Both methods assume input feature independence and linearity between inputs and baselines, offering a rigorous framework for attribution.
Beyond these core attribution methods, Captum also includes perturbation-based techniques such as Feature Ablation, Feature Permutation, and Occlusion. These methods involve modifying or masking input features to measure their impact on the model’s output. Feature Permutation, on the other hand, shuffles feature values within a batch to assess their contribution, though it requires multiple examples for meaningful results.
To ensure the reliability of explanations generated by these methods, Captum provides metrics like Infidelity and Sensitivity. These metrics help users gauge the trustworthiness of the explanations, ensuring that the insights drawn from Captum are both accurate and actionable.
Use Cases
Captum’s versatility makes it applicable across a broad spectrum of domains, particularly in AI security, where understanding model behavior is critical. In computer vision, for example, Captum can be used to explain decisions made by object detection or segmentation models.
In natural language processing, Captum supports the attribution of features in large language models (LLMs). Techniques such as Integrated Gradients or DeepLIFT can be applied to token-level inputs to understand how specific words or phrases influence the model’s output. For instance, if an LLM generates an incorrect or biased response, Captum can help trace the root cause to particular input features, enabling targeted improvements to the model or its training data.
Captum is also instrumental in research settings, where it serves as a benchmarking tool for new interpretability algorithms. Researchers can implement and test their methods within Captum’s framework, comparing performance against established baselines. This collaborative approach accelerates innovation in the field, as contributions from the community are integrated into the library. For example, a researcher developing a novel neuron attribution method could use Captum to validate its effectiveness across different model architectures and datasets.
Key Features and Capabilities
Captum’s core strength lies in its comprehensive suite of attribution algorithms, each designed to address specific interpretability challenges. Integrated Gradients, for instance, provides a theoretically grounded approach to feature attribution by integrating gradients along a path from a baseline to the input. This method is particularly useful for models where input features are continuous, such as images or sensor data, as it captures the cumulative effect of feature changes on the output. The implementation supports both Riemann Sum and Gauss Legendre quadrature rules for approximating the integral, offering flexibility in computational trade-offs.
Gradient SHAP is another key feature, combining the efficiency of gradient-based methods with the theoretical rigor of Shapley values. By adding Gaussian noise to input samples and computing gradients along random paths, Gradient SHAP approximates the average contribution of each feature to the model’s output.
DeepLIFT and its extension, DeepLIFT SHAP, are tailored for understanding non-linear relationships within neural networks. DeepLIFT attributes changes in output to differences between input features and a reference baseline, using multipliers to propagate these changes backward through the network. The Rescale Rule, currently supported in Captum, simplifies this process by normalizing contributions based on the reference context. DeepLIFT SHAP further refines this approach by averaging attributions over a distribution of baselines, providing a more robust estimate of feature importance.
For convolutional neural networks, Captum offers specialized methods like Guided Backpropagation and Deconvolution, which modify the backpropagation process to highlight relevant input regions. Guided Backpropagation ensures that only positive gradients are propagated backward, effectively suppressing noise and focusing on meaningful activations. Deconvolution extends this idea by applying ReLU functions to output gradients, providing a clearer visualization of feature importance in convolutional layers. Guided GradCAM combines these techniques with GradCAM’s class-discriminative capabilities, offering a high-resolution explanation of model decisions in vision tasks.
Perturbation-based methods in Captum, such as Feature Ablation, Feature Permutation, and Occlusion, provide alternative approaches to attribution by directly manipulating input features. Feature Ablation measures the impact of removing or masking features, while Feature Permutation assesses the effect of shuffling feature values within a batch. Occlusion, often referred to as the grey-box or sliding window method, systematically occludes regions of the input to evaluate their contribution. These methods are particularly useful for spatial data like images, where local feature correlations are strong.
Captum’s extensibility is another defining feature, as it welcomes contributions from the research community. Users can implement new attribution algorithms or extend existing ones, ensuring that the library remains at the forefront of interpretability research. This collaborative model has led to the inclusion of diverse methods, from gradient-based approaches to perturbation techniques, catering to a wide range of use cases.
To support the evaluation of attribution methods, Captum includes metrics like Infidelity and Sensitivity. Sensitivity measures how attributions change in response to small input perturbations, indicating the stability of the explanations.