Incident Overview
On June 27, 2024, JFrog disclosed a critical vulnerability in Vanna.AI, a Python-based text-to-SQL library that integrates large language models (LLMs) with database querying capabilities. The vulnerability, identified as CVE-2024-5565, stems from a prompt injection flaw that enables remote code execution (RCE) through dynamically generated Plotly visualization code. This incident serves as a canonical example of how unvalidated LLM-generated code can lead to severe security risks when executed in an application context.
The attack chain exploits Vanna.AI’s default visualization feature, which uses an LLM to generate Python code for rendering query results as charts via Plotly. By crafting a malicious user question, an attacker can manipulate the LLM’s output to include arbitrary Python commands within the generated Plotly code. These commands are subsequently executed via Python’s exec() function, bypassing intended safeguards and achieving full RCE.
The vulnerability was independently discovered by Tong Liu on the Huntr bug bounty platform, who reported it as CVE-2024-5826 after JFrog’s initial disclosure. This dual discovery underscores the growing recognition of prompt injection as a critical attack vector in LLM-driven systems.
Vanna.AI’s architecture relies on Retrieval-Augmented Generation (RAG) to enhance SQL query accuracy, but the integration of LLM-generated code for visualization introduces a dangerous execution path. The library’s default behavior—automatically generating and executing Plotly code—creates a direct link between user input and system commands, amplifying the impact of prompt injection.
The attack surface is broad, as Vanna.AI supports multiple deployment mechanisms, including Jupyter notebooks, Streamlit applications, Flask web servers, and Slack bots.
Possible Root Cause
The root cause of CVE-2024-5565 lies in the library’s design choice to dynamically generate and execute Python code for visualization purposes without adequate validation or sandboxing. The flow begins with the ask() method, which processes user input to generate SQL queries and, by default, triggers the generate_plotly_code() function. This function constructs a prompt for the LLM, incorporating user-provided elements such as the original question and the generated SQL query.
The prompt sent to the LLM for Plotly code generation includes the following structure:
The following is a pandas DataFrame that contains the results of the query that answers the question the user asked: '{question}'.
The following is information about the resulting pandas DataFrame 'df': {df_metadata}.
Can you generate the Python plotly code to chart the results of the dataframe? Assume the data is in a pandas dataframe called 'df'. If there is only one value in the dataframe, use an Indicator. Respond with only Python code. Do not answer with any explanations -- just the code.
Critical injection points exist in the question and sql parameters, both of which are derived from user input. The question parameter is the original user prompt, while the sql parameter is the SQL query generated by the LLM in response to that prompt. By manipulating either of these inputs, an attacker can alter the LLM’s output to include malicious Python code.
While this SQL query is syntactically valid, it embeds a command that, when propagated into the Plotly code generation prompt, could result in the LLM producing Python code executing arbitrary system commands. The generated Plotly code is then passed to Python’s exec() function, which runs it without further scrutiny.
The vulnerability arises because the library’s pre-prompting safeguards, designed to constrain LLM behavior, are ineffective against indirect prompt injection. Unlike direct injection—where user input directly manipulates the LLM’s prompt—indirect injection occurs when user input influences the LLM’s output through intermediate steps, such as SQL query generation. In this case, the SQL query itself becomes a vector for injecting malicious instructions into the subsequent Plotly code generation prompt.
Additionally, the LLM’s SQL generation prompt lacks strict constraints on output format or content, allowing attackers to embed executable payloads within seemingly benign queries.
The absence of code validation or execution restrictions in the visualization pipeline further exacerbates the risk. The library assumes that LLM-generated Plotly code is safe to execute, failing to account for the possibility of adversarial input manipulation.
Takeaways
This incident highlights the dangers of executing LLM-generated code without rigorous validation or containment measures. The case demonstrates how prompt injection can escalate from a theoretical risk to a practical exploit when LLMs are integrated with actionable systems, such as database query execution and visualization tools.
Key lessons for developers and organizations integrating LLMs into their systems include:
- Avoid executing LLM-generated code: Dynamically generated code, even for seemingly benign purposes like visualization, should never be executed directly.
- Implement strict input validation: User input should be sanitized at every stage of processing, particularly when it influences LLM prompts or system commands. This includes validating SQL queries and generated code for adherence to expected formats and safety rules.
- Enhance pre-prompting safeguards: Pre-prompt instructions should explicitly prohibit the generation of executable code or commands. However, developers must recognize that these safeguards can be bypassed through indirect injection vectors, necessitating additional layers of defense.
- Educate on prompt injection risks: Developers must understand that prompt injection is not limited to direct manipulation of LLM prompts.
Vanna.AI has since released hardening guidance and introduced sandboxing options to mitigate this vulnerability. As organizations increasingly rely on LLM-driven tools, the need for robust validation, containment, and monitoring will become paramount.
The canonical nature of this vulnerability—where prompt injection leads to RCE—serves as a warning to the AI community. It illustrates that even well-intentioned features, such as automated visualization, can become attack vectors if not carefully secured.