How DeepTeam Works
DeepTeam is an open-source red teaming framework designed for LLM systems, simulating adversarial attacks to uncover vulnerabilities in AI applications, agents, and RAG pipelines. It operates by dynamically generating adversarial inputs—such as jailbreaks, prompt injections, and multi-turn exploitation attempts—to probe for weaknesses like bias, PII leakage, or SQL injection. Unlike traditional evaluation frameworks, DeepTeam does not rely on pre-prepared datasets; instead, it dynamically creates attacks tailored to the vulnerabilities being tested.
The framework leverages LLMs as judges to evaluate responses, producing binary pass/fail scores with reasoning. DeepTeam integrates with DeepEval, an LLM evaluation framework, to provide structured, reproducible risk assessments.
DeepTeam’s architecture is modular, allowing users to combine vulnerabilities and attack methods to build custom security testing pipelines. It supports both single-turn and multi-turn attacks, with techniques like jailbreaking, prompt injection, and encoding-based obfuscation. The framework also includes guardrails to prevent vulnerabilities in production environments, ensuring that red teaming efforts translate into actionable security improvements.
At its core, DeepTeam functions by wrapping the target LLM system via a model_callback, which generates responses to adversarial inputs. These inputs are dynamically generated based on selected vulnerabilities and attack methods. The responses are then evaluated using LLM-as-a-Judge metrics specific to each vulnerability type, producing binary scores (0 or 1) along with reasoning.
The framework’s design prioritizes flexibility and integration. Users can define custom vulnerabilities or attacks, align testing with industry standards like OWASP Top 10 for LLMs or NIST AI RMF, and run assessments via CLI or Python.
DeepTeam’s evaluation process begins with the generation of baseline attacks for each vulnerability type. These attacks are then used to probe the target LLM system, with responses evaluated against predefined criteria. The framework distinguishes between baseline attacks—simple probes—and adversarial attacks, which enhance baseline attacks using techniques like jailbreaking or multi-turn exploitation. This distinction ensures that testing covers both straightforward and sophisticated attack vectors.
For example, testing for bias vulnerabilities involves generating prompts that probe for stereotypes or unfair treatment across gender, race, or religion. The LLM’s response is then evaluated to determine if it exhibits biased behavior. Similarly, PII leakage tests simulate attempts to extract sensitive personal information, with responses scored based on whether they disclose such data. This structured approach ensures that vulnerabilities are identified and quantified with precision.
Each vulnerability type is associated with a specific metric, such as toxicity detection or bias assessment, which is applied to the LLM’s output. The framework supports custom metrics for user-defined vulnerabilities, allowing organizations to tailor testing to their specific risks. This flexibility is particularly useful for addressing niche or emerging threats that may not be covered by default vulnerabilities.
The framework’s modularity is evident in its support for both single-turn and multi-turn attacks. Single-turn attacks, such as prompt injection or leetspeak (symbolic character substitution), are designed to bypass restrictions in a single interaction. Multi-turn attacks, like linear or tree jailbreaking, involve iterative or parallel refinement of attacks to exploit vulnerabilities over extended conversations.
DeepTeam’s alignment with industry frameworks is a key feature, enabling users to map their red teaming efforts against standards like OWASP Top 10 for LLMs or MITRE ATLAS. The framework also supports integration with safety benchmark datasets like BeaverTails and Aegis, providing curated adversarial test cases for comprehensive testing.
In practice, DeepTeam can be used to test a wide range of LLM systems, from chatbots to autonomous agents. For instance, testing a RAG pipeline might focus on vulnerabilities like context poisoning or cross-context retrieval, where the system’s ability to isolate data is compromised. Similarly, testing an AI agent might involve probing for excessive agency or tool orchestration abuse, where the agent acts beyond its intended scope. The framework’s adaptability makes it suitable for diverse use cases, from pre-deployment testing to ongoing monitoring in production.
Users can define YAML configurations to specify target models, attack types, and evaluation parameters, or use Python to programmatically define vulnerabilities and attacks. This flexibility makes DeepTeam a versatile tool for teams seeking to automate and streamline their LLM security testing processes.
Use Cases
DeepTeam is primarily used for red teaming LLM applications and agents to identify vulnerabilities before deployment or during production monitoring. Its ability to simulate adversarial attacks across categories like bias, toxicity, PII leakage, and prompt injection makes it ideal for teams focused on AI security. The framework is particularly valuable for organizations developing AI-driven products, where security and compliance are critical.
One key use case is pre-deployment testing, where DeepTeam helps identify and mitigate vulnerabilities in LLM systems before they reach users. By simulating attacks like jailbreaking or prompt injection, teams can assess the robustness of their models and implement guardrails to prevent exploitation. This proactive approach reduces the risk of security incidents post-deployment, ensuring that AI systems meet safety and compliance standards.
DeepTeam is also used for ongoing monitoring in production environments. Teams can integrate the framework into their CI/CD pipelines to continuously test LLM systems for emerging vulnerabilities. For example, a chatbot deployed in a customer-facing role can be regularly assessed for bias, toxicity, or PII leakage, with results used to refine the system’s guardrails. This iterative testing ensures that security improvements are tracked over time, aligning with the framework’s stateful testing capabilities.
DeepTeam’s alignment with industry frameworks like OWASP Top 10 for LLMs and NIST AI RMF allows organizations to demonstrate adherence to regulatory requirements. This is particularly useful for sectors like finance or healthcare, where AI systems must comply with strict data privacy and security regulations.
DeepTeam’s support for agentic red teaming makes it valuable for testing autonomous AI agents. DeepTeam’s specialized attack methods, such as authority spoofing or context injection, help identify these risks before they manifest in production.
The framework is also useful for research and development teams exploring new attack vectors or vulnerabilities. By defining custom vulnerabilities and attacks, researchers can test hypotheses about LLM weaknesses or evaluate the effectiveness of new security measures. DeepTeam’s modular architecture allows for experimentation with novel attack techniques, contributing to the broader understanding of AI security risks. This makes it a valuable tool for both academic and industry research.
By defining reusable configurations for vulnerabilities and attacks, teams can ensure consistency in their red teaming efforts.
For developers and security teams, DeepTeam offers a practical way to incorporate red teaming into their workflows. Whether testing a simple chatbot or a complex AI agent, DeepTeam provides the tools necessary to identify and address vulnerabilities systematically.
Key Features and Capabilities
DeepTeam includes over 50 pre-built vulnerabilities spanning categories such as Responsible AI, Data Privacy, Security, Safety, Business, and Agentic risks. Each vulnerability is designed to probe for specific weaknesses, such as bias, toxicity, or PII leakage, with attacks dynamically generated to exploit these vulnerabilities.
DeepTeam’s integration with industry frameworks is a standout feature. It automatically maps vulnerabilities and attacks to standards like OWASP Top 10 for LLMs, OWASP Top 10 for Agents, NIST AI RMF, and MITRE ATLAS. The framework also supports safety benchmark datasets like BeaverTails and Aegis, providing curated test cases for comprehensive evaluation.
The framework includes seven production-ready guardrails designed for real-time classification of LLM inputs and outputs. These guardrails help prevent vulnerabilities from reaching production environments, ensuring that security measures are proactive rather than reactive.
DeepTeam’s modular architecture allows users to build custom security testing pipelines by combining vulnerabilities and attack methods. This flexibility is enhanced by the framework’s support for both CLI and Python-based testing.
These include risks like goal redirection, excessive agency, or tool orchestration abuse, where agents act beyond their intended scope.
DeepTeam’s evaluation process relies on LLM-as-a-Judge metrics to assess responses to adversarial inputs. Each vulnerability type is associated with a specific metric, such as toxicity detection or bias assessment, which produces binary scores (0 or 1) along with reasoning. Users can also define custom metrics for user-defined vulnerabilities, allowing for tailored evaluation criteria.
DeepTeam’s stateful testing capabilities allow users to track security improvements over time. This is particularly useful for organizations that undergo continuous development and deployment cycles, where maintaining a secure posture requires ongoing vigilance. The framework’s ability to save results locally as JSON further supports this use case, providing a historical record of security assessments.
Users can test their systems using models from providers like OpenAI, Anthropic, or Google, or integrate custom models via the model_callback interface.
The framework provides detailed guides on defining vulnerabilities, attacks, and configurations, along with examples for common use cases. Users can also join the Confident AI Discord community for discussions on LLM security, help with attack selection, or general feedback.