How it Works
RobustVLM introduces an unsupervised adversarial fine-tuning method for CLIP vision encoders, improving their resistance to visual adversarial attacks without requiring labeled data. The approach, termed FARE (Fine-tuning Adversarially via Randomized Embeddings), fine-tunes CLIP models on ImageNet for two epochs using an adversarial training strategy with an ℓ∞ norm constraint. Unlike supervised methods such as TeCoA, FARE does not rely on class labels, making it more scalable and efficient. The fine-tuning process focuses on enhancing the robustness of the vision encoder while preserving clean accuracy, which is critical for maintaining performance on non-adversarial inputs.
These perturbations are constrained by an ε value, defining the maximum allowed distortion (e.g., ε = 2/255 or 4/255). The method ensures that the model learns to generalize better under adversarial conditions, as demonstrated by its superior performance in zero-shot classification tasks under attack scenarios. The fine-tuned vision encoders can then be integrated into larger vision-language models (VLMs) like OpenFlamingo and LLaVA without requiring retraining of the entire model.
The core innovation lies in the unsupervised nature of the fine-tuning process. By avoiding the need for labeled data, FARE reduces computational overhead and broadens applicability. The method is evaluated on ViT-L/14 and base-sized CLIP models, with results showing significant improvements in adversarial robustness across various vision-language tasks, including COCO and TextVQA benchmarks.
Use Cases
RobustVLM is designed primarily for testing and red teaming large vision-language models, where adversarial robustness is a critical concern. Its applications include evaluating the resilience of VLMs to visual perturbations, which can simulate real-world attack scenarios such as adversarial patches or noise injections. Researchers and practitioners can use the fine-tuned CLIP models to assess how well their systems maintain performance under adversarial conditions, identifying vulnerabilities that could be exploited in malicious contexts.
By integrating the robust vision encoders into VLMs like OpenFlamingo and LLaVA, users can benchmark the models' ability to handle adversarial inputs while preserving their accuracy on clean data. This is essential for deploying VLMs in safety-critical applications, such as autonomous systems or medical imaging, where adversarial attacks could lead to catastrophic failures.
Additionally, RobustVLM serves as a tool for developing perceptual metrics in computer vision. The adversarially robust CLIP models demonstrate strong performance as metrics for evaluating image similarity and quality, both in clean and adversarial settings. This capability is valuable for applications requiring high fidelity in visual representations, such as content moderation or generative AI systems where adversarial robustness is a priority.
Key Features & Capabilities
The models are categorized into two series: FARE and TeCoA. FARE models are trained unsupervised using the ℓ∞ norm constraint, while TeCoA models employ supervised adversarial fine-tuning with ImageNet labels. This distinction allows users to choose between methods based on their specific requirements for clean accuracy and adversarial robustness.
The repository includes ViT-L/14 models fine-tuned for both FARE and TeCoA, each approximately 1.1 GB in size. These models are available for direct integration into existing CLIP-based pipelines. For example, users can replace the vision encoder of a pre-trained CLIP model with one of the fine-tuned checkpoints to enhance its adversarial resistance. The provided code snippets demonstrate how to load these models using PyTorch and OpenCLIP, ensuring compatibility with standard workflows.
The table of results highlights the trade-offs between clean accuracy and adversarial robustness across different architectures.
The training process for TeCoA and FARE models is documented, with scripts provided for reproducibility. For example, the TeCoA 4 model can be trained using a command specifying the CLIP model name, pretrained weights, and ImageNet dataset path.