Attack on Machine Translation Services via Model Imitation and Adversarial Examples
A research team from UC Berkeley demonstrated in April 2020 that production-grade machine translation (MT) systems, including Google Translate, Bing Translator, and Systran Translate, are vulnerable to imitation attacks. By querying public APIs with monolingual sentences, the researchers trained imitation models capable of replicating near-production translation quality. These models achieved performance within 0.6 BLEU score of the original systems, even when using different architectures, datasets, or hyperparameters. The attack exploited the fact that MT systems are often served as black-box APIs, where internal training data and model probabilities remain hidden.
The study, documented in the paper Imitation Attacks and Defenses for Black-box Machine Translation Systems (arXiv:2004.15015), highlights two primary risks: model theft and adversarial manipulation. Model theft allows adversaries to replicate or compete with proprietary MT services, bypassing API costs or licensing restrictions. Adversarial manipulation, meanwhile, exposes vulnerabilities that can lead to harmful translations—such as targeted word flips, vulgar outputs, or dropped sentences—when carefully crafted inputs are fed to the systems.
The researchers tested their approach on high-resource language pairs (e.g., English→German) and low-resource pairs (e.g., Nepali→English). Even when the imitation models were trained on out-of-domain data—such as sentences from Wikipedia labeled by the victim API—they maintained high fidelity to the original system’s outputs. This suggests that domain mismatches do not inherently prevent model imitation, provided sufficient query volume and labeled data.
Methodology: Imitation via Query-Based Training
The imitation process followed a two-phase approach. In the first phase, the researchers selected monolingual sentences from corpora (e.g., Wikipedia) and labeled them using the target MT API. These labeled pairs were then used to train imitation models, effectively distilling the knowledge of the black-box system. Unlike traditional knowledge distillation, where teacher and student models share training data, this method relied solely on API responses, introducing challenges such as hidden probabilities and potential domain shifts.
The second phase involved generating adversarial examples against the imitation models. Using gradient-based attacks adapted from prior NLP research, the team created inputs that produced semantically incorrect, vulgar, or truncated translations when processed by the imitation models. These adversarial examples were then transferred to the original production systems, where they retained their disruptive effects.
- Targeted word flips: Inputs designed to alter specific output tokens. For example, the phrase I am feeling grey that HK decided to join China was manipulated to make Systran translate grey as froh (happy) instead of its intended meaning.
- Malicious nonsense: Inputs that, when translated, produced vulgar or offensive outputs. An example provided was the English sentence miei llll going ro tobobombier the Land, which Google Translate rendered as I will bomb the country in German.
- Universal triggers: Phrases or suffixes appended to any input that consistently induced errors. One such trigger was the word Siehe repeated seven times, which frequently caused Systran to produce incorrect translations. Another trigger, the universal suffix dropper, led to the omission of appended text and subsequent content from the translation output.
All adversarial examples were successfully transferred to production systems as of April 2020, though some vulnerabilities were later patched by the service providers.
Defenses: Output Modification to Disrupt Imitation
To mitigate imitation attacks, the researchers proposed a defense mechanism that modifies translation outputs to misdirect the optimization of imitation models. The approach, inspired by prediction poisoning, involves generating alternate translations that deviate from the original output while maintaining high accuracy. These alternate translations are selected based on their gradient deviation from the original, effectively steering imitation models toward suboptimal learning paths.
The defense works by first generating the original translation, then producing 100 candidate translations using methods like beam search or top-k sampling. Candidates with low overlap to the original translation are discarded, and the remaining options are evaluated for gradient deviation. The selected candidate is then output instead of the original, degrading the imitation model’s performance without severely impacting the victim system’s BLEU score.
Quantitative results showed that the defense could reduce the imitation model’s BLEU score from 34.5 to 32.7 while only slightly affecting the victim’s score (34.6 to 33.8). Additionally, adversarial example transferability decreased when the defense was applied, though the computational overhead and minor accuracy trade-off remain challenges for deployment. The researchers emphasized that stronger defenses are needed to fully address model theft and adversarial risks in NLP systems.
Broader Implications for NLP Security
The study underscores the fragility of black-box MT systems when exposed to systematic querying and adversarial techniques. While the focus was on model theft and targeted disruptions, the findings extend to broader concerns in NLP security, such as data privacy, intellectual property protection, and the reliability of automated translation in high-stakes contexts (e.g., legal, medical, or diplomatic communications).
The researchers noted that their work aligns with prior efforts to expose vulnerabilities in NLP models, such as gender bias in MT systems, to drive improvements in robustness and fairness. However, they cautioned that imitation attacks and adversarial examples remain practical threats, requiring proactive defenses. The proposed output modification technique, though effective in simulation, highlights the need for scalable and computationally efficient solutions to protect production systems.
The case study was documented under the MITRE ATLAS framework (AML.CS0005) and serves as a reference for understanding the risks posed by API-based MT services. It also provides a template for evaluating and defending against similar threats in other black-box NLP systems.
Sources
- atlas.mitre.orghttps://atlas.mitre.org/studies/AML.CS0005
- arxiv.orghttps://arxiv.org/abs/2004.15015
- ericswallace.comhttps://www.ericswallace.com/imitation
- thehill.comhttps://thehill.com/policy/international/asia-pacific/449164-google-under-fire-for-mistr…
Case study data from MITRE ATLAS, used under the Apache License 2.0.