How Purple Llama Works
Purple Llama is an umbrella project designed to provide a comprehensive framework for trust and safety in generative AI systems. It integrates multiple tools and evaluation benchmarks to address risks associated with large language models (LLMs) in cybersecurity, input/output moderation, and secure code generation. The project adopts a purple teaming approach, combining offensive (red team) and defensive (blue team) strategies to collaboratively evaluate and mitigate risks.
The project is structured around five core components: Llama Guard for input/output classification, Prompt Guard for detecting malicious prompt injections, LlamaFirewall for securing agent workflows, Code Shield for filtering insecure generated code, and CyberSecEval benchmarks for assessing cybersecurity risks in LLMs.
Llama Guard is a series of high-performance models fine-tuned on Meta-Llama 3.1 and 3.2 architectures. These models are optimized to detect violations across the MLCommons standard hazards taxonomy, including cybersecurity threats and unsafe outputs. They support the latest Llama 3.2 capabilities, such as expanded language support (7 new languages), a 128k context window, and image reasoning. Llama Guard 3 models are specifically trained to identify malicious code outputs and prevent execution in environments using code interpreters.
Prompt Guard focuses on protecting LLM-powered applications from prompt injection attacks and jailbreaking attempts. Prompt injections occur when untrusted data is introduced into the model’s context window to manipulate its behavior, while jailbreaks are malicious instructions designed to bypass built-in safety features. Prompt Guard acts as a preemptive layer to detect and block such threats before they compromise system integrity.
Code Shield addresses the risk of insecure code generation by LLMs. It provides inference-time filtering to mitigate vulnerabilities such as code interpreter abuse and insecure command execution. This tool is particularly critical for applications where LLMs interact with code interpreters or execute system commands.
LlamaFirewall is tailored for securing agent workflows in AI systems.
The CyberSecEval benchmarks serve as industry-wide standards for evaluating LLM cybersecurity risks. Developed in collaboration with security experts and aligned with frameworks like CWE and MITRE ATT&CK, these benchmarks provide metrics to quantify risks such as insecure code suggestions and compliance with malicious requests. CyberSecEval has evolved through three versions, expanding to include visual prompt injection tests, spear phishing assessments, and evaluations of autonomous offensive cyber operations.
CyberSecEval v1 was the first of its kind, offering tools to measure LLM vulnerabilities in generating malicious code or aiding cyberattacks. Version 2 introduced additional tests for code interpreter abuse and offensive cybersecurity capabilities, while version 3 added visual prompt injection and spear phishing evaluations. These benchmarks are integrated into platforms like Papers With Code and HELM, ensuring transparency and reproducibility in safety assessments.
All components within Purple Llama are designed for permissive licensing to encourage community collaboration. Evals and benchmarks are released under the MIT license, while safeguard models like Llama Guard and Prompt Guard follow the Llama Community License.
The project is part of Meta’s broader commitment to open and responsible AI development. It aligns with initiatives like the AI Alliance and the White House’s responsible AI commitments, fostering a collaborative ecosystem where developers can adopt best practices for secure AI deployment. Purple Llama’s tools are integrated into the Llama reference system, providing a standardized safety layer for developers building with open generative AI models.
Use Cases
Llama Guard is ideal for applications requiring real-time input/output moderation, such as customer support chatbots, content moderation platforms, or any system where user-generated content must be screened for harmful or unsafe outputs.
Prompt Guard is essential for securing applications exposed to untrusted inputs, such as public APIs, collaborative editing tools, or multi-user AI assistants. By detecting prompt injections and jailbreaking attempts, it prevents adversarial manipulation of LLM behavior, ensuring that the system remains resilient against malicious intent.
Code Shield is particularly valuable in development environments where LLMs assist with coding tasks. It prevents the generation of insecure or exploitable code, making it suitable for integrated development environments (IDEs), automated testing frameworks, or any workflow where code safety is a priority. This tool is also critical for systems that execute user-generated code, such as sandboxed interpreters or automated deployment pipelines.
LlamaFirewall is designed for AI agents operating in autonomous or semi-autonomous workflows.
The CyberSecEval benchmarks are useful for researchers, security auditors, and organizations evaluating the cybersecurity posture of their LLMs. These tools provide objective metrics to assess risks such as code vulnerability suggestions, compliance with malicious prompts, and susceptibility to prompt injection attacks. They enable organizations to benchmark their models against industry standards and identify areas for improvement.
Key Features & Capabilities
- Llama Guard: A suite of input/output classifiers fine-tuned on Meta-Llama 3 models, optimized for detecting violations across the MLCommons hazards taxonomy. Supports 7 new languages, a 128k context window, and image reasoning. Specifically trained to identify malicious code outputs and prevent execution in code interpreter environments.
- Prompt Guard: Detects prompt injection attacks and jailbreaking attempts to protect LLM-powered applications from adversarial manipulation. Blocks untrusted inputs that exploit context windows to execute unintended instructions or override safety features.
- Code Shield: Provides inference-time filtering for insecure code generated by LLMs. Mitigates risks such as code interpreter abuse, insecure command execution, and vulnerable code suggestions.
- LlamaFirewall: Secures agent workflows by monitoring and controlling interactions with external systems.
- CyberSecEval Benchmarks: Industry-standard evaluations for assessing LLM cybersecurity risks. Includes metrics for insecure code suggestions, compliance with malicious requests, and susceptibility to prompt injection. Three versions cover visual prompt injection, spear phishing, and autonomous offensive cyber operations.
- Permissive Licensing: Components are licensed under MIT (for evals and Code Shield) or the Llama Community License (for safeguard models), enabling broad adoption in research and commercial applications. Facilitates collaboration and standardization in the AI safety ecosystem.
- Integration with Llama Reference System: Tools are designed to integrate seamlessly with Meta’s Llama models, providing a standardized safety layer for developers. Resources for deployment and customization are available in the Llama-recipe GitHub repository.
- Collaboration with Industry Partners: Developed in partnership with organizations such as the AI Alliance, MLCommons, Hugging Face, and major cloud providers (AWS, Google Cloud, Microsoft).
- Alignment with Responsible AI Guidelines: Reflects Meta’s Responsible Use Guide recommendations, emphasizing the need for input/output filtering and adherence to content guidelines. Supports compliance with regulatory and ethical standards in AI deployment.