Home / Case Studies
Case Studies
How organisations put AI security into practice: the situation, the approach and the measured outcome. Cases are anonymised and some details are composited to protect the organisations involved.
| Record | Industry | Programme | Country |
|---|---|---|---|
| Achieving Code Execution in MathGPT via Prompt InjectionThe publicly available Streamlit application MathGPT uses GPT-3, a large language model (LLM), to answer user-generated math questions. Recent… | AI developers and research labs | Prompt injection defence design | Germany |
| Advai: Advanced Evaluation of AI-Powered Identity Verification SystemsAdvai evaluated identity-verification vendors' AI systems with a method that goes beyond testing on sample image datasets, giving buyers evidence… | Payments and fintech | Third-party and vendor risk | United Kingdom |
| Advai: Assurance of Computer Vision AI in the Security IndustryAdvai's toolkit was applied to an object-detection model used in the physical security industry, assessing its performance, security and… | Cybersecurity vendors | Model risk management and validation | United Kingdom |
| Advai: Implementing a Risk-Driven AI Regulatory Compliance FrameworkAdvai's compliance framework applies ISO 31000 risk-management principles to AI: risks from each AI system are assessed and mitigated in a way… | Cross-industry | Governance operating model | United Kingdom |
| Advai: Operational Boundaries Calibration for AI Systems via Adversarial Robustness TechniquesAdvai stress-tests enterprise AI systems with adversarial methods to establish their fault tolerances, so operational boundaries can be set from… | Cross-industry | Red teaming programme | United Kingdom |
| Advai: Regulatory Aligned AI Deployment FrameworkAdvai's deployment framework for high-risk domains treats regulatory risk, robustness and societal impact as deployment criteria alongside… | Cross-industry | Preparing for a regulation | United Kingdom |
| Advai: Robustness Assessment of a Facial Verification System Against Adversarial AttacksAdvai assessed a facial verification system used for authentication, testing its resilience to image manipulation and adversarial attacks intended… | Payments and fintech | Red teaming programme | United Kingdom |
| Advai: Robustness Assurance Framework for Guardrail Implementation in Large Language Models (LLMs)Advai applies a robustness assurance framework that attacks a client LLM with adversarial prompts optimised across several model architectures,… | AI developers and research labs | Red teaming programme | United Kingdom |
| Advai: Streamlining AI Governance with Advai Insight for Enhanced Robustness, Risk Management and ComplianceAdvai Insight is an enterprise platform that turns AI risk and robustness measurements into actionable views for non-technical stakeholders,… | Cross-industry | Governance operating model | United Kingdom |
| AI ClickFix: Hijacking Computer-Use Agents Using ClickFixEmbrace the Red demonstrated that AI computer-use agents are vulnerable to social engineering attacks and can be manipulated into executing… | AI developers and research labs | Agent security architecture | Global |
| AIKatz: Attacking LLM Desktop ApplicationsResearchers at Lumia have demonstrated that it is possible to extract authentication tokens from the memory of LLM Desktop Applications. An… | Software supply chain | Red teaming programme | Israel |
| AI Model Tampering via Supply Chain AttackResearchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container… | Software supply chain | Model supply chain assurance | Japan |
| Anthropic disrupts GTG-1002, an AI-orchestrated cyber-espionage campaignA China-attributed group jailbroke an agentic coding tool to run reconnaissance, exploitation, lateral movement and exfiltration largely… | AI developers and research labs | Incident response and recovery | United States |
| Anthropic's August 2025 threat report documents disrupted cyber and fraud operationsAnthropic's casebook of misuse it disrupted, including an AI-coding-agent data-extortion scheme, employment fraud by North Korean operators,… | AI developers and research labs | Incident response and recovery | United States |
| Anthropic's September 2026 report: GTG-20006 and blocked biological-misuse attemptsAnthropic's fourth threat report covers December 2025 to August 2026, including a Russia-linked espionage campaign against drone manufacturers and… | AI developers and research labs | Incident response and recovery | United States |
| Arbitrary Code Execution with Google ColabGoogle Colab is a Jupyter Notebook service that executes on virtual machines. Jupyter Notebooks are often used for ML and data science research… | Cloud and infrastructure | Red teaming programme | United States |
| Attack on Machine Translation ServicesMachine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research… | Consumer apps and social media | Red teaming programme | United States |
| Attempted Evasion of ML Phishing Webpage Detection SystemAdversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their… | Cybersecurity vendors | Incident response and recovery | Global |
| AWS certifies four AI services to ISO/IEC 42001 and passes its first surveillance auditAWS obtained accredited ISO/IEC 42001 certification of its AI management system for four named services, describing itself as the first major… | Cloud and infrastructure | Audit and certification | United States |
| Backdoor Attack on Deep Learning Models in Mobile AppsDeep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that… | Technology | Model supply chain assurance | Global |
| Universal bank replaces standing agent credentials with just-in-time, task-scoped accessAn internal audit of 210 agent deployments found most running with service accounts broader than any human's; the bank rebuilt agent access on… | Banking | Agent identity and access | North America |
| BCG joins the first hundred organisations certified to ISO/IEC 42001BCG certified its AI management system to ISO/IEC 42001, citing controls, accountability and transparency across its AI work. | Professional services | Audit and certification | United States |
| Botnet Domain Generation Algorithm (DGA) Detection EvasionThe Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA)… | Cybersecurity vendors | Red teaming programme | United States |
| Bypassing Cylance's AI Malware DetectionResearchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a… | Cybersecurity vendors | Red teaming programme | United States |
| Bypassing ID.me Identity VerificationAn individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's… | Public sector | Incident response and recovery | United States |
| Camera Hijack Attack on Facial Recognition SystemThis type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems… | Public sector | Incident response and recovery | China |
| ChatGPT Conversation ExfiltrationEmbrace the Red demonstrated that ChatGPT users' conversations can be exfiltrated via an indirect prompt injection. To execute the attack, a… | AI developers and research labs | Agent security architecture | United States |
| ChatGPT Package HallucinationResearchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package… | AI developers and research labs | Red teaming programme | Israel |
| ClearBank: Safeguarding generative AI use cases in a regulated fintech banking APIClearBank runs retrieval-augmented LLM use cases inside its banking API (enquiries monitoring, a policy-review bot and client screening) on its… | Payments and fintech | Generative AI application controls | United Kingdom |
| ClearviewAI MisconfigurationClearview AI makes a facial recognition tool that searches publicly available photos for matches. This tool has been used for investigative… | Cloud and infrastructure | Incident response and recovery | United States |
| Code to Deploy Destructive AI Agent Discovered in Amazon Q VS Code ExtensionOn July 13th, 2025, a malicious actor using the GitHub username "lkmanka58" used an inappropriately scoped GitHub token to make a commit… | Software supply chain | Agent security architecture | United States |
| Compromised PyTorch Dependency ChainLinux packages for PyTorch's pre-release version, called Pytorch-nightly, were compromised from December 25 to 30, 2022 by a malicious binary… | Software supply chain | Model supply chain assurance | United States |
| Confusing Antimalware Neural NetworksCloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built… | Cybersecurity vendors | Red teaming programme | Russia |
| Credo AI Governance Platform: Reinsurance Provider Algorithmic Bias Assessment and ReportingA global reinsurance provider used Credo AI's governance platform to produce standardised algorithmic bias reports for new regulatory requirements… | Insurance | Governance operating model | Global |
| Credo AI Policy Packs: Human Resources Startup compliance with NYC LL-144An HR technology start-up used Credo AI's policy pack for NYC Local Law 144, which turns the law's requirements into actionable controls, to… | Professional services | Preparing for a regulation | United States |
| Credo AI Transparency Reports: Facial Recognition ApplicationA facial recognition service provider used Credo AI's platform to publish transparency reports on the fairness and performance of its identity… | Technology | Customer assurance | United States |
| Data Destruction via Indirect Prompt Injection Targeting Claude Computer-UseSecurity researchers at HiddenLayer demonstrated that an indirect prompt injection targeting Claude’s Computer Use AI can lead to execution of… | AI developers and research labs | Prompt injection defence design | United States |
| Data Exfiltration from Slack AI via Indirect Prompt InjectionPromptArmor demonstrated that private data can be exfiltrated from Slack AI via indirect prompt injections. The attack relied on Slack AI… | Cybersecurity vendors | Prompt injection defence design | United States |
| Data Exfiltration via Agent Tools in Copilot StudioResearchers from Zenity demonstrated how an organization’s data can be exfiltrated via prompt injections that target an AI-powered customer… | Software supply chain | Prompt injection defence design | Israel |
| Data Exfiltration via an MCP Server used by CursorThe Backslash Security Research Team demonstrated that a Model Context Protocol (MCP) tool can be used as a vector for an indirect prompt… | Software supply chain | Agent security architecture | United States |
| Data Exfiltration via Remote Poisoned MCP ToolResearchers at Invariant Labs demonstrated that AI agents configured with remote Model Context Protocol (MCP) Tools can be vulnerable to model… | Software supply chain | Agent security architecture | Switzerland |
| Evasion of Deep Learning Detector for Malware C&C TrafficThe Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP… | Cybersecurity vendors | Red teaming programme | United States |
| Exposed ClawdBot Control Interfaces Leads to Credential Access and ExecutionA security researcher identified hundreds of exposed ClawdBot control interfaces on the public internet. ClawdBot (now OpenClaw) “is a personal AI… | AI developers and research labs | Agent security architecture | Global |
| Face Identification System Evasion via Physical CountermeasuresMITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a… | Technology | Red teaming programme | United States |
| FairNow: Conversational AI and Chatbot Bias AssessmentFairNow's chatbot bias assessment lets deployers test a conversational system by generating realistic, representative prompts of the kind users… | Cross-industry | Continuous evaluation and monitoring | United States |
| FairNow: NYC Bias Audit With Synthetic Data (NYC Local Law 144)For New York City Local Law 144, in force since July 2023 and the first US law to require bias audits of AI used in hiring and promotion, FairNow… | Cross-industry | Audit and certification | United States |
| FairNow: Regulatory Compliance Implementation and the NIST AI RMF / ISO ReadinessFairNow's platform identifies which standards and laws apply to an organisation's AI use, then manages the activities needed to comply, covering… | Cross-industry | Preparing for a regulation | United States |
| Financial Transaction Hijacking with M365 Copilot as an InsiderResearchers from Zenity conducted a red teaming exercise in August 2024 that successfully manipulated Microsoft 365 Copilot.[<sup>\[1\]</sup>][1]… | Payments and fintech | Agent security architecture | United States |