How PromptMe Works
PromptMe is a vulnerable-by-design application designed to simulate real-world security risks in large language models (LLMs) and their web integrations. It functions as a Capture The Flag (CTF) platform, where users engage with 10 hands-on challenges directly inspired by the OWASP Top 10 for Large Language Model Applications. Each challenge is structured to demonstrate how vulnerabilities can be discovered and exploited, providing practical insights into securing LLM-based systems.
The project is built using Python and the Ollama framework, which supports open-source LLM models. Users interact with the application through a web interface hosted locally at http://127.0.0.1:5000. The challenges are accessible via a start button corresponding to each category, such as LLM01, and are designed to be solved step-by-step, with optional hints and flags awarded upon completion.
The application intentionally includes insecure code and dependencies to highlight potential risks. Users are encouraged to explore vulnerabilities in a controlled environment, such as a virtual machine or sandbox, to understand the implications of insecure implementations without exposing production systems to harm.
Setup requires Python 3.10 or higher, the Ollama framework, and specific LLM models (e.g., Mistral, Llama3, SQLCoder, and Granite3.1-moe). Once dependencies are installed, running python main.py initializes the application, allowing users to begin the challenges immediately.
The project is currently compatible with macOS and Linux systems, with Windows support planned for future updates.
Use Cases
PromptMe is primarily intended for AI security professionals, researchers, and educators seeking to deepen their understanding of LLM vulnerabilities. By engaging with the challenges, users can develop skills in identifying, analyzing, and mitigating security risks in LLM applications, aligning with the OWASP LLM Top 10 framework.
The CTF-style challenges serve as an interactive learning tool, allowing users to apply theoretical knowledge in a practical setting. Each challenge is designed to reflect common vulnerabilities found in real-world LLM deployments, such as prompt injection, data leakage, and model manipulation. This hands-on approach helps users recognize patterns and implement defensive strategies effectively.
Additionally, PromptMe can be used in educational settings, such as workshops or training programs, to teach students about LLM security. Instructors can leverage the challenges to demonstrate attack vectors and mitigation techniques, fostering a proactive security mindset among learners.
For red teaming and penetration testing, PromptMe provides a controlled environment to simulate adversarial scenarios.
Key Features & Capabilities
- OWASP LLM Top 10 Alignment: The 10 challenges directly map to the OWASP Top 10 for Large Language Model Applications, covering critical vulnerabilities like prompt injection, model denial-of-service, and sensitive information disclosure. This alignment ensures users gain exposure to the most relevant and up-to-date threats in LLM security.
- Interactive CTF Challenges: Each challenge is self-contained with clear objectives, optional hints, and a flag system to validate successful exploitation.
- No API Key Requirement: PromptMe eliminates the need for external API keys, simplifying setup and reducing barriers to entry. Users can focus solely on the challenges without managing third-party dependencies or rate limits.
- Open-Source LLM Models: The project utilizes open-source models hosted via the Ollama framework, including Mistral, Llama3, and Granite3.1-moe. This ensures reproducibility and transparency, allowing users to inspect the underlying models and their behaviors during exploitation.
- Sandboxed Environment: The application is designed to be run in isolated environments, such as virtual machines or Docker containers. This precaution prevents accidental exposure of vulnerabilities to production systems and adheres to ethical security research practices.
- Beginner-Friendly Solutions: Detailed solutions are provided for each challenge, making the platform accessible to users with varying levels of expertise. Beginners can follow step-by-step guides to understand the vulnerabilities, while advanced users can explore alternative exploitation paths.
- Community Contributions: The project welcomes contributions from the community, including bug fixes, documentation improvements, and new challenge additions.
- Cross-Platform Compatibility: While primarily supported on macOS and Linux, the project is actively developing Windows compatibility. This expansion will broaden accessibility for users across different operating systems.
The project is licensed under the MIT License, ensuring broad permissibility for educational and research use. It is maintained by a team of developers, including Divyesh, Srithesh, Praveen, Ranjit, and Sumanth, who continue to refine the platform based on user feedback and evolving security landscapes.