The Problem & Context
Large Language Models (LLMs) have rapidly transitioned from research tools to foundational components in real-world applications, enabling functionalities such as interactive chat, document summarization, and API-driven actions. The integration of LLMs into systems like Bing’s GPT-4-powered Chat, Microsoft 365 Copilot, and third-party plugins has accelerated their adoption, with platforms like ChatGPT reaching over 100 million users within months of launch. However, this rapid integration has outpaced the development of security safeguards, exposing applications to novel attack vectors.
Traditional prompt injection (PI) attacks assume direct interaction with the LLM, where adversaries manipulate inputs to override instructions or bypass controls. These attacks have been documented in black-box settings, even with mitigations in place. Yet, the rise of LLM-integrated applications introduces a critical blind spot: the blurring of boundaries between data and instructions. Unlike direct PI, indirect prompt injection (IPI) exploits this ambiguity by embedding malicious prompts within data that the LLM retrieves during inference. If ingested, these prompts can remotely compromise the model’s behavior, affecting not only the attacker but also benign users interacting with the same system.
The implications of IPI extend beyond individual applications. Retrieved prompts can act as arbitrary code, enabling adversaries to manipulate functionality, control API calls, or execute unauthorized actions. Recent incidents have already demonstrated unintended behaviors triggered by retrieved data, such as hostile responses. IPI formalizes and expands on these risks, revealing a broader threat landscape where adversaries can achieve persistent compromise, data theft, denial of service, and even large-scale disinformation campaigns. The autonomous capabilities of modern LLMs—such as self-prompting, tool integration, and adaptive planning—further amplify these risks, as models may autonomously execute attacker-defined goals with minimal oversight.
Existing research on LLM security has primarily focused on direct prompt manipulation, adversarial training, or input sanitization. However, IPI represents an uncharted attack surface, requiring a systematic taxonomy to classify vulnerabilities, delivery mechanisms, and potential impacts. This paper addresses this gap by introducing the first comprehensive framework for analyzing IPI threats in LLM-integrated applications, grounded in computer security principles.
Methodology
The study employs a multi-faceted approach to investigate IPI, combining theoretical analysis, empirical validation, and taxonomy development.
First, the researchers develop a taxonomy to categorize IPI threats based on their delivery mechanisms, target systems, and potential consequences. This framework distinguishes between direct and indirect attack vectors, highlighting how retrieved data can serve as a medium for adversarial control. The taxonomy also differentiates between passive and active exploitation scenarios, where passive attacks involve data contamination (e.g., worming or ecosystem poisoning), and active attacks involve real-time manipulation of LLM behavior during inference.
Second, the practical feasibility of IPI is demonstrated through experiments on both real-world and synthetic LLM-integrated applications. Real-world targets include Bing’s GPT-4-powered Chat and code-completion engines, while synthetic applications are constructed using GPT-4 to simulate diverse use cases. Attack scenarios are designed to test specific vulnerabilities, such as arbitrary code execution via retrieved prompts, unauthorized API calls, and functionality manipulation. The paper provides reproducible attack prompts and a public repository (GitHub: greshake/llm-security) to facilitate further research.
Third, the study evaluates the broader implications of IPI by analyzing its alignment with traditional security principles, such as remote control, persistence, and privilege escalation. The researchers also explore novel threats enabled by LLM autonomy, including adaptive disinformation dissemination and user manipulation. By comparing IPI to established attack vectors in software and cybersecurity, the paper underscores its potential to redefine threat landscapes in AI-driven systems.
The methodology relies on black-box testing to assess vulnerabilities without prior knowledge of internal model architectures, mirroring real-world adversarial conditions. This approach ensures that findings are applicable to both proprietary and open-source LLM systems, regardless of their underlying training or deployment mechanisms.
Key Takeaways
Indirect prompt injection (IPI) introduces a critical vulnerability in LLM-integrated applications by exploiting the indistinguishability between data and instructions during retrieval-augmented inference. Unlike traditional prompt injection, IPI does not require direct access to the LLM; instead, adversaries embed malicious prompts within data sources that the model processes at runtime. This attack vector can compromise applications remotely, affecting millions of users who interact with contaminated data.
- Arbitrary Code Execution via Prompts: Retrieved prompts can function as executable instructions, enabling adversaries to manipulate LLM behavior, bypass safety controls, and trigger unauthorized actions. For example, an IPI attack could instruct a code-completion tool to generate malicious scripts or a chatbot to disclose sensitive information.
- Systemic Compromise and Persistence: IPI can lead to persistent compromise of LLM-integrated systems, analogous to traditional malware. Adversaries may inject prompts into widely accessed data repositories (e.g., public documents, APIs, or third-party plugins), ensuring prolonged exposure. This includes scenarios like worming, where infected data propagates across interconnected systems, or information ecosystem contamination, where retrieved prompts alter the model’s responses globally.
- Autonomous Agent Exploitation: The rise of autonomous AI agents—systems that combine LLMs with tool integration, memory, and adaptive planning—expands the attack surface for IPI. Agents equipped with retrieval mechanisms may autonomously execute attacker-defined goals, such as spreading disinformation, automating phishing campaigns, or manipulating user interactions. The lack of human oversight in these systems exacerbates the risks posed by IPI.
- Real-World Feasibility: The paper demonstrates IPI attacks against live systems, including Bing’s GPT-4 Chat and code-completion tools, proving their viability in production environments. Synthetic applications built on GPT-4 further validate the attack’s generality across different use cases. These experiments highlight the urgent need for defenses tailored to indirect attack vectors.
- Taxonomy and Defense Gaps: The proposed taxonomy categorizes IPI threats by delivery method (e.g., data poisoning, API manipulation), target (e.g., chatbots, code generators, autonomous agents), and impact (e.g., data theft, denial of service, disinformation). However, the study identifies a critical gap in existing defenses: most mitigation strategies focus on direct prompt sanitization or input validation, which are ineffective against IPI.
- Broader Security Implications: IPI challenges fundamental assumptions in LLM security, particularly the distinction between user-provided and system-retrieved inputs. The paper emphasizes the need for collaborative efforts between researchers, developers, and policymakers to establish robust security standards for LLM-integrated applications.
The findings underscore that IPI is not a theoretical risk but a practical threat with measurable consequences. Without targeted defenses, the rapid proliferation of LLM applications could inadvertently create large-scale vulnerabilities, undermining trust in AI systems and exposing users to unprecedented security risks.