Red teaming AI systems has matured from prompt tinkering into a discipline with tooling, benchmarks and regulatory standing: the EU AI Act's systemic-risk obligations, the code of practice, California SB 53 and the frontier safety frameworks all require adversarial testing, and buyers increasingly ask for evidence of it. The difficulty is scope. A model, an application built on it and an agent that acts through it have different attack surfaces, and a test that covers one says little about the others.
Three layers, three questions
At the model layer the question is what the model will do when asked: refusal robustness against jailbreaks, dangerous capability elicitation, training data extraction, and behaviour under fine-tuning. Tools are HarmBench and JailbreakBench for comparable numbers, garak for breadth, PyRIT and DeepTeam for adaptive multi-turn campaigns, and extraction probes with canaries. At the application layer the question is what an attacker can make the application do through its inputs: prompt injection through every field and document, retrieval boundary failures across tenants, output handling into sinks, hidden context exposure, unbounded consumption, and the ordinary web security of the surrounding platform, which is where McHire, DeepSeek and the reported McKinsey case failed. At the agent layer the question is what an attacker can make the agent do with its tools and credentials: goal hijack through untrusted content, tool misuse, privilege abuse, MCP supply chain, code execution, memory poisoning and inter-agent propagation. AgentDojo provides the benchmark structure, promptfoo and DeepTeam the OWASP-keyed test generation, and a live environment with real tools and fake data provides the truth.
Method
Start with the threat model, not the tool. Enumerate the data the system can reach, the sources that can write into its context and the sinks its output reaches; the lethal trifecta tells you where the serious paths are. Map the attack classes from NIST AI 100-2, MITRE ATLAS and the OWASP lists onto those paths. Then test in tiers. Breadth first: automated scans across the attack classes to find the easy failures and establish a baseline. Depth second: adaptive campaigns in which an attacker model or a human adjusts to what the defences do, including multi-turn escalation, encoding and translation, payloads written for human readers, and injection through every content source the system consumes. Realism third: run the agent with its real tools against a staged environment and attempt to reach the sinks, because a finding that the model can be persuaded matters only if persuasion reaches something. Include the platform: authentication, authorisation on every object, exposed services, CI pipelines and the MCP servers in use.
Two methodological rules from the research literature apply. The attacker moves second, so evaluate defences against attacks adapted to them, and report the utility cost of each defence alongside its effect. And models recognise tests, so vary framing, use realistic affordances and treat a clean result as a hypothesis.
Frontier and dangerous capabilities
For organisations testing models for cyber, biological or autonomy capabilities, the summer 2026 incidents rewrote the rules. Run without safeguards only inside hardened sandboxes with no internet access except the model API; validate the sandbox by having the model attempt escape under supervision; confirm tasks are solvable by the intended route; state scope as instructions, not as descriptions of the environment; monitor in real time with authority to halt; and treat the evaluation environment as production infrastructure. The UK AI Security Institute's incident report and Anthropic's evaluator best practices are the reference documents.
Evidence
A red team report that says the model was jailbroken is not evidence for a release decision. Evidence looks like this: the threat model and the attack classes covered, with the ones excluded and why; for each class, the method, the number of attempts, the attack success rate and the utility cost of any defence tested; the transcripts of successful attacks, so that engineers can reproduce them; the controls that stopped attacks and the ones that did not; and a residual risk statement. Convert findings into regression tests in promptfoo or DeepTeam so that the next change cannot silently reopen them, and into detections so that the same attack in production is seen. Map the report to the framework the audience uses, whether OWASP, ATLAS, NIST AI 600-1 or the EU AI Act's Article 15 attack classes.
Cadence and governance
Test before release and on every material change to the model, the prompt, the tools or the retrieval corpus. Run breadth scans continuously in CI, adaptive campaigns quarterly, and a full engagement with external testers annually or before a major expansion of autonomy. Keep the red team independent of the team that built the system, give them the design documents, and have findings owned by an engineering lead with a deadline. For frontier-scale systems, add external evaluation by a safety institute or an independent lab, as the EU code of practice and the 2026 executive order's framework expect.
Common failures of AI red teams
- Testing the model and calling it a system test.
- Static prompt lists against a defence that was trained on them.
- Ignoring the platform: default credentials and exposed databases have caused more AI breaches than clever prompts.
- No utility measurement, so defences that break the product look secure.
- Findings that stay in a PDF instead of becoming tests and detections.
- Cyber evaluations with internet access and no live monitoring.
Done properly, red teaming is the mechanism by which an AI security programme learns. Done as theatre, it is the mechanism by which it convinces itself.
