How it Works
Foolbox is a Python library designed to execute adversarial attacks on machine learning models, particularly deep neural networks. Unlike earlier versions, Foolbox 3.0 was rebuilt from scratch using EagerPy, a framework that enables native performance across PyTorch, TensorFlow, and JAX without requiring separate codebases for each. This redesign eliminates redundancy and ensures compatibility with all three frameworks under a single implementation.
For example, a PyTorch model can be integrated into Foolbox using fb.PyTorchModel(model, bounds=(0, 1)), where bounds specify the valid input range for the model.
Foolbox supports both gradient-based and decision-based adversarial attacks. Gradient-based attacks, such as the Projected Gradient Descent (PGD) attack, manipulate input data by exploiting gradients computed during model training. Decision-based attacks, on the other hand, rely solely on the model’s predictions without accessing gradients, making them suitable for black-box scenarios where internal model details are unavailable.
The library includes extensive type annotations, which help catch bugs early by enforcing type consistency before execution.
Performance is a key focus of Foolbox 3.0. The transition to EagerPy has significantly improved speed compared to previous versions, as demonstrated in performance benchmarks provided in the repository.
Use Cases
Foolbox is primarily used by researchers and practitioners to benchmark the robustness of machine learning models against adversarial attacks. By systematically applying a variety of attacks, users can identify vulnerabilities in models and assess their resilience to malicious inputs.
The library supports both white-box and black-box attack scenarios. In white-box settings, attackers have full access to the model’s architecture and gradients, allowing for highly targeted attacks like PGD or Fast Gradient Sign Method (FGSM). In black-box settings, where only input-output pairs are observable, Foolbox provides decision-based attacks that infer model behavior through iterative queries.
Researchers can use Foolbox to generate adversarial examples with varying perturbation magnitudes (epsilon values) and measure the success rate of attacks across different perturbation budgets. This process helps quantify how susceptible the model is to adversarial perturbations and guides the development of more robust defenses.
Foolbox’s compatibility with PyTorch, TensorFlow, and JAX makes it versatile for cross-framework comparisons. Researchers can test the same adversarial attacks on models implemented in different frameworks, ensuring fairness in robustness evaluations.
The library also facilitates the integration of new adversarial attacks. Contributors can extend Foolbox by implementing additional attack methods, such as the recently added Momentum Iterative Fast Gradient Sign Method (MIFGSM) or Adam-based PGD attacks.
Key Features & Capabilities
- Native Framework Support: Foolbox 3.0 is built on EagerPy, enabling seamless integration with PyTorch, TensorFlow, and JAX. Users can define models in their preferred framework and interact with them through Foolbox’s unified interface.
- Comprehensive Attack Library: The library includes a wide range of gradient-based attacks (e.g., LinfPGD, FGSM, BIM) and decision-based attacks (e.g., boundary attacks, hop-skip-jump attacks). These attacks cover both white-box and black-box scenarios, providing flexibility for different evaluation needs.
- Type Safety and Bug Prevention: Extensive type annotations in Foolbox help detect errors during development by enforcing type consistency.
- This prevents attacks from generating invalid inputs, which could skew robustness evaluations.
- Performance Optimizations: The redesign of Foolbox 3.0 using EagerPy has led to significant speed improvements over previous versions. The library is regularly updated to maintain compatibility with newer versions of PyTorch, TensorFlow, and JAX.
- Modular and Extensible Design: Foolbox encourages contributions from the research community. Users can add new adversarial attacks, fix bugs, or improve existing functionality.
- Dependencies like PyTorch, TensorFlow, and JAX are not bundled with Foolbox to avoid conflicts with different CUDA versions or system architectures. Users must install these frameworks separately based on their specific requirements.