How It Works
Agent Smith introduces a novel attack mechanism targeting multimodal large language model (MLLM) agents, demonstrating how a single adversarial image can trigger an infectious jailbreak across an entire multi-agent system. The attack exploits the shared memory and interaction patterns of agents to propagate harmful behaviors exponentially. The core concept revolves around the idea that once an adversarial image infects a single agent, it spreads through pairwise chat interactions, causing unaligned behaviors in nearly all agents within approximately 27 to 31 chat rounds in a system of one million LLaVA-1.5 agents.
The attack leverages two primary components: a border attack and a pixel attack, both designed to craft adversarial images that evade detection while maximizing jailbreak success. The border attack adjusts perturbations within a defined budget, while the pixel attack modifies pixel values directly. These adversarial images are then introduced into the memory of a randomly selected agent, initiating the spread. The system simulates randomized pairwise chats, where agents exchange images and text histories, enabling the adversarial image to propagate through questioning and answering interactions.
The propagation mechanism relies on three key probabilities: the chance of a virus-carrying questioning agent transmitting the adversarial image (denoted as β), the probability of a virus-carrying agent exhibiting jailbroken symptoms (αQ), and the probability of an answering agent becoming infected after receiving the adversarial image (αA). The study finds that failures in infection often stem from low α values during the chat process, indicating that the adversarial image must effectively trigger harmful behaviors in agents to ensure widespread propagation.
Theoretical analysis in the paper derives a principle for evaluating defense mechanisms against infectious jailbreaks, though practical defenses remain an open challenge. The framework formalizes the dynamics of infection in ideal conditions, providing a foundation for understanding how adversarial images can exploit multi-agent interactions to achieve systemic compromise.
Use Cases
Agent Smith is primarily designed for testing and red teaming MLLM agents in multi-agent environments. Its key application lies in evaluating the resilience of large-scale agent systems against adversarial attacks, particularly those that exploit shared memory and interaction patterns. Researchers and security practitioners can use Agent Smith to simulate infectious jailbreak scenarios, assess the effectiveness of existing defenses, and identify vulnerabilities in multi-agent architectures.
The tool enables the generation of adversarial images tailored to specific perturbation budgets and textual diversity levels, allowing for controlled experiments in both high and low diversity chat scenarios. This flexibility makes it suitable for benchmarking the robustness of MLLM agents under adversarial conditions, as well as for exploring the limits of current defense strategies. By simulating environments with up to one million agents, Agent Smith provides a scalable approach to studying systemic risks in distributed AI systems.
Another critical use case is the evaluation of defense mechanisms against infectious jailbreaks. The paper’s theoretical framework offers a basis for testing whether proposed defenses can provably restrain the spread of adversarial images.
This capability is valuable for understanding the temporal evolution of adversarial propagation and identifying critical points where interventions could limit damage. By providing reproducible experiments and configurable attack parameters, Agent Smith facilitates collaborative research in AI security.
Key Features & Capabilities
Agent Smith is implemented in Python and requires specific hardware and software dependencies to function. The setup involves cloning the repository, creating a Python environment with version 3.10, and installing dependencies such as PyTorch 2.1.0, Transformers, Accelerate, and custom libraries like Diff-JPEG for adversarial image optimization.
The attack module includes pre-generated benign chat records for 64 LLaVA-1.5 7B agents under high and low diversity scenarios, stored in simulationhigh.csv and simulationlow.csv. Users can regenerate these records or use the provided data to craft adversarial images. The attack process employs Fully Sharded Data Parallel (FSDP) via the Accelerate library to optimize performance, with configurable parameters for perturbation budgets, textual diversity, and image augmentation.
Two primary attack methods are supported: the border attack and the pixel attack. The border attack adjusts image perturbations within a defined budget, while the pixel attack modifies pixel values directly, with ε values ranging from 1 to 255. Image augmentation can be enabled through parameters such as random flipping, resizing, and JPEG compression, allowing for more robust adversarial image generation.
Validation of crafted adversarial images requires matching the parameters used during the attack phase. The validate.py script generates an adversarial image named advimage.png in the experimental folder, which can then be used to evaluate infection rates in multi-agent simulations. The simulation module supports two main workflows: generating ensemble records for adversarial image crafting and evaluating infectious jailbreak propagation across up to 256 agents in a batch process.
The simulationbatch.py script generates pairwise chat records for high or low diversity scenarios, while simulationtestbatch.py evaluates the spread of adversarial images in a controlled environment.