ProofPoint Evasion

ProofPoint Evasion via Adversarial Machine Learning: The Proof Pudding Case

In September 2019, researchers from Silent Break Security demonstrated a method to bypass ProofPoint’s email protection system by exploiting its machine learning-based scoring mechanism. The attack, documented under CVE-2019-20634 and referred to as Proof Pudding, involved creating a functional replica of ProofPoint’s email classification model. This replica was then used to analyze and manipulate malicious emails, allowing them to evade detection by the live system. The core vulnerability stemmed from ProofPoint’s reliance on a numerical scoring system, where each word in an email is assigned a value based on multiple variables. If the cumulative score falls below a threshold, the email is flagged as spam. By reverse-engineering the scoring logic, researchers crafted emails that achieved higher scores, bypassing the spam filter entirely.

The attack was presented at DerbyCon 2019 by Will Pearce and Nick Landers, with the full methodology and code repository made publicly available on GitHub. The project, titled Proof-Pudding, includes stolen pre-scored datasets, pre-trained models, and extracted insights from the research. The repository serves as both a case study and a practical tool for understanding how adversarial techniques can be applied to evade machine learning-driven security systems.

Technical Breakdown of the Attack

The Proof Pudding methodology relies on several key steps, beginning with data acquisition. Researchers obtained pre-scored datasets from ProofPoint’s system, which contained emails labeled with their respective scores (ranging from 1 to 999, where higher values indicate safer content). These datasets were used to train a copy-cat model using Python 3, Keras, and TensorFlow, with an Artificial Neural Network (ANN) architecture combined with Bag of Words tokenization. The goal was to replicate ProofPoint’s scoring behavior as closely as possible.

Training the model involved minimizing the loss function, specifically targeting the mlxlogscore metric. The performance of the trained models was measured using Mean Absolute Error (MAE), which quantifies the average deviation between predicted and actual scores. The researchers provided pre-trained models to simplify replication, allowing others to test the attack without extensive computational resources. For example, training a model on link-based data could be executed with the command:

python proofpudding.py train -d links ./models/my_link_model.h5

Once trained, the model could score new emails or links to predict their likelihood of evasion. For instance, scoring an email text would use a command like:

python proofpudding.py score -m ./models/texts.h5 email.text

Beyond scoring, the researchers developed a technique to reverse-engineer the model’s decision-making process. This involved analyzing how the removal or alteration of specific words (tokens) affected the overall score. By toggling tokens from 1 to 0 and observing the score changes, they identified which words contributed most significantly to higher or lower scores. These insights were compiled into lists of ‘good’ and ‘bad’ tokens, such as:

These token lists provided actionable guidance for crafting malicious emails. The researchers also noted that ProofPoint’s scoring models are likely fine-tuned versions of a broader, general model rather than entirely unique models for each client. This observation suggests that the attack could be adapted across different organizational contexts with minimal adjustments.

Implications for AI Security in Email Protection

The Proof Pudding case highlights critical vulnerabilities in machine learning-based security systems, particularly those relying on static scoring mechanisms. By demonstrating that an adversary could replicate and manipulate a proprietary model, the researchers exposed flaws in ProofPoint’s reliance on numerical thresholds for spam detection. The attack underscores the need for dynamic, adaptive security measures that account for adversarial manipulation.

One of the most significant findings from this research is the reproducibility of the attack. The public availability of the Proof Pudding repository lowers the barrier for others to test similar evasion techniques against their own email security systems.

Additionally, the case study serves as a practical example of how adversarial machine learning can be weaponized against enterprise security infrastructure. The use of pre-trained models and token analysis demonstrates that even without direct access to the original model’s architecture, an attacker can infer critical decision-making patterns.

The Proof Pudding attack also sheds light on the limitations of traditional machine learning approaches in cybersecurity. Static scoring systems, while efficient, are susceptible to reverse-engineering and manipulation.

Key Takeaways for AI Security Professionals

• Model Replicability is a Risk: The ability to replicate proprietary models using stolen datasets demonstrates that even encrypted or obfuscated systems can be reverse-engineered.
• Token Analysis Can Expose Weaknesses: The identification of high-impact tokens (e.g., ‘gmt’, ‘plugins’) reveals how small changes in input data can significantly alter model outputs.
• Pre-Trained Models Lower the Barrier to Attack: The availability of pre-trained models in the Proof Pudding repository simplifies the process of testing evasion tactics.
• Adversarial Training is Essential: Incorporating adversarial training into ML security models can improve their resilience against manipulation. By exposing models to adversarial examples during development, organizations can identify and mitigate vulnerabilities before deployment.
• Dynamic Scoring Alone is Insufficient: Relying solely on static scoring mechanisms, such as ProofPoint’s numerical thresholds, is insufficient for long-term security.
• Client-Specific Models May Not Be Unique: The observation that ProofPoint likely uses fine-tuned versions of a general model rather than entirely unique models per client suggests that evasion techniques could be broadly applicable.

The Proof Pudding case remains a pivotal example of how adversarial machine learning can compromise AI-driven security systems. By understanding the mechanics of this attack, security professionals can better prepare for similar threats and develop more robust defenses against evolving adversarial techniques.

Sources

• atlas.mitre.orghttps://atlas.mitre.org/studies/AML.CS0008
• nvd.nist.govhttps://nvd.nist.gov/vuln/detail/CVE-2019-20634
• github.comhttps://github.com/moohax/Talks/blob/master/slides/DerbyCon19.pdf
• github.comhttps://github.com/moohax/Proof-Pudding
• youtube.comhttps://www.youtube.com/watch?v=CsvkYoxtexQ&ab-channel=AdrianCrenshaw

Case study data from MITRE ATLAS, used under the Apache License 2.0.