What is Garak?
Garak is an open-source LLM vulnerability scanner designed to identify weaknesses in large language models (LLMs) and dialog systems. Developed by NVIDIA and released in 2023 under the Apache 2.0 license, Garak systematically probes models for vulnerabilities such as prompt injection, jailbreaks, data leakage, toxicity generation, hallucinations, and misinformation. Its purpose is to assist security teams and model builders in assessing deployments before release, ensuring that unintended behaviors or security flaws are detected early. Garak operates similarly to tools like nmap or Metasploit Framework but is specialized for LLMs, combining static, dynamic, and adaptive probes to explore failure modes in language models.
The tool is implemented in Python and supports a wide range of models, including those hosted on Hugging Face Hub, Replicate, OpenAI API, AWS Bedrock, Litellm, and custom REST endpoints. Garak’s flexibility extends to both chat and completion models, making it a versatile choice for red-teaming and security assessments across diverse AI deployments.
How Garak Works
Garak functions as a command-line tool that automates the process of probing LLMs for vulnerabilities. Users specify a target model or endpoint using command-line arguments, and Garak executes a series of predefined attack probes against it. Each probe is designed to test for specific weaknesses, such as encoding-based prompt injection, jailbreak prompts (e.g., DAN variants), data leakage, or toxicity generation. The tool generates responses from the model and evaluates them against detectors that identify undesirable behaviors.
By default, Garak runs all available probes against a specified model. However, users can narrow the scope by selecting specific probe families or individual plugins. For example, running garak --spec probes.promptinject will test only for prompt injection vulnerabilities. Results are displayed in real-time via progress bars, with detailed logs saved in JSONL format for further analysis. Errors are logged separately in garak.log, while a basic analysis script (analyse/analyselog.py) helps identify the most critical vulnerabilities from the generated data.
Key Features and Capabilities
- Comprehensive Probe Library: Garak includes hundreds of attack probes categorized into families like prompt injection, jailbreaks, data leakage, and toxicity. Each probe is paired with detectors that evaluate model responses for vulnerabilities. For instance, the probes.encoding module tests for encoding-based injection attacks, while probes.dan.Dan110 probes for DAN jailbreak prompts. Users can explore all available probes using garak --listprobes.
- Multi-Model Support: Garak supports a broad range of LLM interfaces, including Hugging Face (local and API-based), OpenAI, Replicate, Cohere, Groq, AWS Bedrock, and custom REST endpoints. For Hugging Face models, users can specify the model name directly (e.g., RWKV/rwkv-4-169m-pile) or use private endpoints with API tokens. Garak also supports NVIDIA’s NIM platform and locally run models via ggml.
- Dynamic and Adaptive Testing: Garak combines static, dynamic, and adaptive probes to simulate real-world attack scenarios. For example, Garak can iterate through multiple generations per prompt (default: 10) to increase the likelihood of detecting weaknesses.
- Detailed Reporting and Logging: Results are presented in a structured format, showing whether a probe succeeded or failed. Failures are marked with a rate indicating how many generations exhibited undesirable behavior. Garak also supports JSON and file output modes for qualitative reviews, enhancing usability for teams conducting thorough security assessments.
- Community-Driven Development: Garak is actively maintained by NVIDIA and the broader AI security community. Users can contribute by submitting pull requests, reporting issues, or suggesting new probes. The project’s documentation, available at docs.garak.ai, includes guides for setup, usage, and troubleshooting. Community engagement is encouraged via Discord and Twitter (@garakllm), where users can share findings, seek help, and collaborate on improvements.
Use Cases for Garak
Garak is primarily used by security teams, AI researchers, and model builders to assess the robustness of LLMs before deployment.
- Pre-Deployment Security Audits: Organizations can use Garak to identify vulnerabilities in custom or third-party models before integrating them into production systems. For example, a team developing a chatbot for customer support might run Garak to ensure the model cannot be exploited for data leakage or prompt injection.
- Red-Teaming Exercises: Security professionals leverage Garak to simulate adversarial attacks on LLMs, testing how models respond to jailbreak prompts, toxic inputs, or encoding-based manipulations. This helps refine model safeguards and improve resilience against real-world threats.
- This is particularly useful for enterprises deploying AI systems in regions with strict compliance frameworks.
- Benchmarking and Model Comparison: Researchers and developers can use Garak to compare the security posture of different models.
- Educational and Research Purposes: Garak serves as a tool for teaching AI security principles, demonstrating how LLMs can be manipulated, and exploring mitigation strategies.
Technical Implementation and Workflow
Users interact with the tool via command-line arguments, specifying the target model, probe selection, and generation parameters.
- Installation: Garak can be installed via pip (python -m pip install -U garak) or directly from the GitHub repository for the latest features. It is compatible with Linux and macOS and can be run in a dedicated Conda environment to manage dependencies.
- Target Specification: Users define the model to scan using --targettype (e.g., huggingface, openai, bedrock) and --targetname (e.g., gpt-5-nano, RWKV/rwkv-4-169m-pile). For cloud-based models, API keys are set via environment variables (e.g., OPENAIAPIKEY, BEDROCKAPIKEY).
- Probe Selection: By default, Garak runs all probes. Users can refine the scope using --spec to target specific families (e.g., probes.promptinject) or individual plugins (e.g., probes.lmrc.SlurUsage). This granularity allows for focused testing based on specific concerns.
- Execution and Logging: Garak generates responses from the model and evaluates them against detectors. Progress is displayed via progress bars, and results are logged in JSONL format. Errors are recorded in garak.log, while the analysis script (analyselog.py) helps prioritize findings.
- Result Interpretation: Output includes failure rates for each probe, indicating how many generations exhibited undesirable behavior. For example, testing OpenAI’s text-babbage-001 with encoding probes might reveal vulnerabilities to quoted-printable and MIME injections, while newer models like GPT-5 may show higher susceptibility across multiple encoding types.
Example Workflows
- Testing OpenAI Models for Encoding Injection: To probe an OpenAI model for encoding-based prompt injection, users set the OPENAIAPIKEY environment variable and run:
export OPENAIAPIKEY="sk-123XXXXXXXXXXXX"
python3 -m garak --targettype openai --targetname gpt-5-nano --spec probes.encoding
This command tests the specified model against encoding-based attacks, with results indicating which encoding methods (e.g., quoted-printable, MIME) successfully bypassed model safeguards.
- Custom REST Endpoint Testing: For models hosted on private REST endpoints, users configure a YAML file to define the endpoint’s API specifications.
- AWS Bedrock Model Assessment: To test an AWS Bedrock model (e.g., claude-3-sonnet), users set the BEDROCKAPIKEY and BEDROCKREGION environment variables and run:
export BEDROCKAPIKEY="your-api-key"
export BEDROCKREGION="us-east-1"
garak --targettype bedrock --targetname claude-3-sonnet --spec probes.dan
This workflow evaluates the model’s susceptibility to jailbreak attempts, aligning with broader security assessments for cloud-based AI services.
Recent Developments and Improvements
- EU AI Act Mapping: Release v0.17.0 added reference tags to categorize probe results according to the EU AI Act’s risk categories.
- Context-Aware Scanning (CAS): Introduced in v0.16.0, CAS enables users to provide custom context and requirements for target expectations. For example, users can define specific intents (e.g., testing for data leakage) and detect attack vectors that exploit edges in model safeguards. The feature is still evolving, with community feedback shaping its future development.
- Output Flexibility: Users can now generate results in JSON or file formats for qualitative reviews, enhancing Garak’s utility in collaborative security assessments.