DeepRobust: A PyTorch Adversarial Library for Image and Graph Models
DeepRobust is an open-source PyTorch library developed by the Data Science and Engineering Lab at Michigan State University. Released in 2019 under the MIT License, it specializes in adversarial machine learning, providing tools for both attacking and defending models in image and graph neural network (GNN) domains. The library is designed to support researchers focused on evaluating and improving the robustness of GNNs, though it also covers image-based adversarial techniques.
Its development aligns with the growing need for standardized benchmarks in adversarial machine learning, particularly for GNNs, where robustness remains a critical challenge.
DeepRobust is implemented in Python and integrates with PyTorch, making it accessible for researchers and practitioners familiar with these tools. Its modular structure allows users to experiment with a wide range of adversarial techniques without needing to implement them from scratch.
The library’s design emphasizes reproducibility and extensibility, with a clear separation between image and graph-related functionalities. Users can leverage its pre-trained models, attack methods, and defense strategies to test the resilience of their own models or benchmark against published results.
How It Works
DeepRobust is organized into two primary packages: one for image-based adversarial methods and another for graph-based methods. Each package includes attack algorithms, defense mechanisms, and utilities for model training, evaluation, and data preprocessing.
Image Package
The image package supports adversarial attacks and defenses for convolutional neural networks (CNNs) and other image models. It includes algorithms such as the Projected Gradient Descent (PGD) attack, which perturbs input images to deceive classifiers, and defense mechanisms like adversarial training (e.g., PGD training). The package also provides tools for preprocessing, such as the APE-GAN method, and supports datasets like MNIST, CIFAR-10, and ImageNet.
- Attack Methods: PGD, DeepFool, and other gradient-based and optimization-based attacks tailored for image data.
- Defense Methods: Adversarial training frameworks (e.g., PGD training) and model architectures designed to resist adversarial perturbations.
- Preprocessing: Techniques like APE-GAN for generating robust training data.
- Benchmarking: Pre-trained models and evaluation scripts to compare attack success rates and defense effectiveness.
The package also includes utilities for downloading models, loading datasets, and generating adversarial examples programmatically. Users can train custom models, apply attacks, and evaluate defenses using provided examples and configurations.
Graph Package
The graph package extends DeepRobust’s capabilities to graph neural networks, addressing the unique challenges of adversarial robustness in non-Euclidean data structures. It supports attacks that manipulate graph structures (e.g., edge additions/deletions) and node features, as well as defenses like robust GNN architectures and graph sanitization techniques.
- Attack Methods: Metattack, PRBCD, and UGBA, which target graph connectivity and node embeddings to degrade model performance.
- Defense Methods: Models like AirGNN and GCNJaccard, designed to mitigate the impact of adversarial perturbations.
- Dataset Support: Compatibility with PyTorch Geometric datasets (e.g., Cora, Citeseer, OGB-Arxiv) and tools for converting between DeepRobust and PyTorch Geometric formats.
- Scalability: Algorithms optimized for large-scale graphs, such as PRBCD for attacking graphs with millions of nodes.
The graph package also includes utilities for loading datasets, setting up surrogate models, and generating adversarial perturbations.
Installation and Environment
It can be installed via pip (pip install deeprobust) or from source using Git. The library provides a lightweight installation option (python setupempty.py install) for users who wish to avoid installing all dependencies upfront.
The documentation and examples are structured to guide users through installation, basic usage, and advanced workflows. For instance, the image package includes scripts for training a CNN on MNIST, applying PGD attacks, and evaluating defenses. Similarly, the graph package offers tutorials for attacking and defending GNNs on datasets like Cora and OGB-Arxiv.
Use Cases
DeepRobust is primarily used by researchers and practitioners in adversarial machine learning, particularly those studying the robustness of GNNs.
- Benchmarking Adversarial Robustness: Researchers can use DeepRobust to evaluate the effectiveness of new defense mechanisms against established attack methods, ensuring their work meets rigorous standards.
- Developing New Attacks and Defenses: The library’s modular design allows users to extend existing algorithms or integrate custom methods, facilitating innovation in adversarial ML.
- Red-Teaming Machine Learning Systems: Organizations can use DeepRobust to simulate real-world adversarial scenarios, identifying vulnerabilities in their models before deployment.
For example, a researcher studying GNN robustness might use DeepRobust to:
- Train a GCN on the Cora dataset.
- Apply the Metattack algorithm to perturb the graph structure.
- Measure the drop in model accuracy to quantify vulnerability.
- Test a defense mechanism like AirGNN and compare its performance against the attack.
Similarly, an image-based adversarial study might involve training a ResNet on CIFAR-10, generating adversarial examples with PGD, and evaluating the model’s resilience under adversarial conditions.
The library’s support for large-scale datasets (e.g., OGB-Arxiv) and scalable attacks (e.g., PRBCD) also makes it suitable for industry applications where robustness is critical, such as fraud detection or recommendation systems.
Key Features and Capabilities
DeepRobust stands out for its comprehensive coverage of adversarial techniques, benchmarking capabilities, and support for both image and graph domains.
- Comprehensive Algorithm Support: The library includes over 20 attack and defense algorithms, covering gradient-based, optimization-based, and heuristic methods. Notable algorithms include:
- Attacks: PGD, DeepFool, Metattack, PRBCD, and UGBA (for backdoor attacks on graphs).
- Defenses: Adversarial training (PGD training), robust architectures (AirGNN, GCNJaccard), and preprocessing techniques (APE-GAN).
- Benchmarking and Reproducibility: DeepRobust provides pre-trained models, standardized evaluation scripts, and published results to ensure reproducibility. Users can compare their work against established baselines, such as those reported in the AAAI 2021 paper.
- Dataset and Model Compatibility: The library supports a wide range of datasets, including MNIST, CIFAR-10, ImageNet (for images), and Cora, Citeseer, and OGB-Arxiv (for graphs).
- Scalability: Recent updates, such as the PRBCD attack and support for large-scale graphs, demonstrate DeepRobust’s ability to handle real-world datasets with millions of nodes and edges.
- Modular and Extensible Design: The library’s codebase is organized into clear modules for attacks, defenses, and utilities.
- Documentation and Examples: DeepRobust includes detailed documentation, tutorials, and example scripts for both image and graph workflows.
- Regular Updates and Community Support: The library is actively maintained, with updates addressing issues like out-of-memory errors (e.g., fixes in version 0.2.9) and adding new features (e.g., UGBA backdoor attacks in 2023). Contributions are welcome, and the maintainers respond to issues promptly.
Notable Updates and Milestones
Since its initial release, DeepRobust has undergone significant evolution, reflecting advancements in adversarial ML research:
- 2020: First public release, with support for basic image and graph attacks/defenses. Introduction of pip installation and documentation.
- 2021: Published at AAAI 2021; added support for ImageNet, OGB datasets, and scalable attacks like MedianGCN. Version 0.2.2 introduced black-box attack subpackages for images.
- 2022–2023: Focus on graph robustness, with additions like the AirGNN defense (NeurIPS 2021), PRBCD attack (NeurIPS 2021), and UGBA backdoor attack (WWW 2023). Version 0.2.9 addressed compatibility issues with newer PyTorch versions.
- 2023: Introduction of lightweight installation options and optimizations for large-scale graph attacks, such as PRBCD on OGB-Arxiv.
Technical Implementation
DeepRobust’s architecture is designed to streamline the process of testing and developing adversarial methods.
Image Workflow
- The library provides utilities like trainmodel.train() to simplify this process.
- Attack Generation: Attacks like PGD are instantiated and applied to input data.
- Evaluation: The library includes scripts to evaluate attack success rates and defense effectiveness, such as evaluationattack.py, which compares adversarial accuracy against clean accuracy.
Example code for generating adversarial images with PGD:
Graph Workflow
- Dataset Loading: Users load graph datasets (e.g., Cora) using DeepRobust’s Dataset class, which handles data splitting and preprocessing.
- Surrogate Model Setup: A surrogate GNN (e.g., GCN) is trained on the clean dataset to simulate a victim model.
- Attack Execution: Attacks like Metattack modify the graph’s adjacency matrix to reduce model accuracy. For instance, Metattack adds or removes edges to create perturbations.
- Defense Application: Users can deploy defenses like AirGNN or GCNJaccard, which are trained to resist adversarial perturbations.
from deeprobust.graph.globalattack import Metattack
model = Metattack(model=surrogate, nnodes=adj.shape[0], featureshape=features.shape, device=device)
model.attack(features, adj, labels, idxtrain, idxunlabeled, perturbations, llconstraint=False)
Benchmarking and Reproducibility
- Published Baselines: Results from peer-reviewed papers (e.g., AAAI 2021) are included, allowing users to compare their work against established metrics.
- Evaluation Scripts: Tools like testtrain.py automate the process of measuring attack success and defense robustness.