Prompt Injection Attack and Defense: A Comprehensive Resource
Prompt injection remains one of the most pressing security concerns for large language models (LLMs) such as ChatGPT. The repository prompt-hacker-collections serves as a structured and detailed resource for researchers, students, and security professionals to explore the mechanics, examples, and countermeasures related to prompt injection attacks. Developed by yunwei37 and maintained since 2023, this open-source project is licensed under the permissive MIT License, ensuring broad accessibility for academic and defensive research purposes.
The collection is organized into distinct sections, each addressing a specific aspect of prompt injection—from foundational concepts to practical attack and defense strategies. Its primary purpose is to provide runtime defense insights, particularly focusing on prompt injection detection and mitigation techniques. The repository supports multiple languages, reflecting its applicability across diverse technical and research communities.
How It Works
The repository is structured to facilitate both study and hands-on experimentation. These materials serve as a foundation for understanding more advanced techniques documented within the collection.
At its core, the repository categorizes prompts into four key areas: prompt reverse engineering, jailbreak prompts, prompt attacks, and prompt defenses. Each category is meticulously documented in YAML format, allowing for easy parsing, analysis, and integration into research or defensive frameworks.
Prompt Collections and Organization
These prompts are designed to illustrate how attackers manipulate LLMs to bypass restrictions, extract sensitive information, or induce unintended behaviors. The collection includes prompts specifically tailored for jailbreaking, a technique used to circumvent model constraints and access restricted functionalities.
One notable example is the DAN (Do Anything Now) prompt, a widely recognized jailbreak technique originally identified by Reddit users. This prompt instructs the model to adopt a persona free from OpenAI’s policy restrictions, enabling responses that include unverified information, real-time data fabrication, and content that would otherwise be censored. The repository provides the full YAML-structured DAN 11.0 prompt, demonstrating its implementation and the dual-response mechanism it employs—one adhering to standard model behavior and the other simulating a jailbroken state.
Beyond jailbreaking, the collection also includes prompt reverse engineering examples, such as those used to dissect the source prompts of tools like Notion AI and GitHub Copilot. These examples highlight how attackers or researchers can analyze and replicate the underlying logic of proprietary prompts, offering insights into model behavior and potential weaknesses. Additionally, the repository covers Midjourney /describe reverse engineering, showcasing how visual prompts can be deconstructed to understand generative AI decision-making processes.
Jailbreak Prompts and Their Mechanisms
Jailbreak prompts are a critical focus of the repository, given their ability to expose vulnerabilities in LLMs. The jailbreak/ directory contains dozens of such prompts, organized to target multiple models and use cases. These prompts often employ psychological manipulation, such as framing the model as a liberated entity or exploiting its tendency to comply with ambiguous instructions. For instance, a jailbreak prompt might instruct the model to ignore its ethical guidelines temporarily, framing the request as a hypothetical scenario or a role-playing exercise.
The repository emphasizes that jailbreaking techniques are not limited to ChatGPT but apply broadly to other LLMs, including those from competitors and open-source projects. Each jailbreak example is documented with clear annotations, explaining its intended effect, the model’s likely response, and the underlying attack vector.
Prompt Attacks and Defensive Strategies
The Prompt Attacks section catalogs techniques used to exploit LLMs through carefully crafted inputs. The repository provides real-world examples of successful attacks, including cases where models were induced to generate harmful content, disclose proprietary information, or perform unauthorized actions.
Complementing the attack examples, the Prompt Defense section offers countermeasures and best practices for mitigating prompt injection risks.
Use Cases
The repository’s content is primarily targeted at academic researchers, AI security professionals, and ethical hackers engaged in red-teaming exercises. Students and educators can use the collection to demonstrate how LLMs can be exploited, the importance of defensive programming, and the ethical considerations surrounding AI security.
For defensive developers, the repository serves as a benchmark for testing the resilience of their models against prompt injection. By analyzing the provided attack examples, developers can identify weaknesses in their own systems and implement targeted defenses.
Structured and Extensible Prompt Database
One of the repository’s standout features is its YAML-based prompt database, which ensures consistency, readability, and ease of integration.
The database is designed to be extensible, encouraging contributions from the broader security community.
Cross-Model Applicability
The repository’s prompts are not limited to a single LLM but are curated to work across multiple platforms, including ChatGPT, Copilot, Midjourney, and other proprietary or open-source models. This cross-model applicability makes the collection a versatile tool for researchers studying the generalizability of prompt injection techniques.
Educational and Ethical Focus
While the repository documents advanced attack techniques, its primary intent is educational and ethical. The disclaimer explicitly states that the project is intended for academic research and defensive purposes only, with no endorsement of malicious use.
By bridging theoretical research and practical defense, the repository supports a holistic approach to prompt injection mitigation.
Related Resources and Community Contributions
The repository maintains a list of related resources to deepen users’ understanding of prompt injection and defensive strategies.
Contributions to the repository are encouraged through GitHub issues and pull requests, with clear contribution guidelines provided to ensure consistency and quality. The project’s permissive MIT License further facilitates adoption, enabling researchers and organizations to adapt the content for their specific needs without legal barriers.