Gray-Box Adversarial Attacks on Cloud-Based Antimalware ML Models
The Kaspersky ML research team investigated the vulnerability of cloud-deployed antimalware machine learning models to adversarial attacks under a gray-box scenario. In this setup, feature extraction occurs on users’ systems, while model inference is handled by cybersecurity company servers. The team demonstrated that knowledge of feature construction alone is sufficient to craft adversarial malware samples capable of evading detection without requiring full model access.
This research builds on broader concerns about the susceptibility of neural networks to adversarial manipulations. While initial studies focused on image recognition—showing that minor pixel-level changes could mislead classifiers—the concept has since expanded to other domains, including malware detection. The Kaspersky team’s work specifically targets the gray-box model, where attackers lack direct access to the model architecture but can infer feature representations and query the system for classification outcomes.
The attack was executed in June 2021 as part of the MITRE ATLAS case study (AML.CS0014), confirming that adversarial techniques can bypass even sophisticated cloud-based ML defenses. The findings underscore a critical gap in security assumptions: even when models are deployed in a cloud environment, feature-level knowledge can be exploited to manipulate detection outcomes.
Methodology and Attack Scenarios
The Kaspersky team evaluated three adversarial attack scenarios, prioritizing white-box and gray-box approaches due to their practical feasibility.
The gray-box attack, however, aligns more closely with real-world deployment models. Here, attackers exploit feature-level knowledge—such as the structure of PE headers, section statistics, and string representations—without direct access to the model. The team replicated this scenario by constructing a simplified neural network mirroring Kaspersky’s antimalware ML pipeline. This model processed static analysis features from portable executable (PE) files, including:
- PE header attributes (file size, overlay size, executable characteristics, imported/exported functions)
- Section metadata (number of sections, physical/virtual sizes, entropy metrics)
- File strings encoded via Bloom filters, a space-efficient probabilistic data structure that maps strings to bit vectors.
The model was trained on a dataset of 300 million files, evenly split between benign and malicious samples. Performance was optimized for a false positive rate (FPR) of 10^-5, achieving approximately 60% true positive detection (TPR) on test malware samples. This threshold reflects Kaspersky’s emphasis on minimizing false positives while maintaining robust detection capabilities.
Adversarial Attack Execution and Results
- Feature perturbation: Adjusting extracted features (e.g., section entropy, string Bloom filter patterns) to misalign with the model’s decision boundary.
- Functionality preservation: Ensuring modifications did not disrupt the malware’s core operations, a critical constraint in adversarial attacks.
- Objective function optimization: Balancing two competing goals: reducing the model’s detection score while minimizing the number of injected bytes to avoid detection via heuristic checks.
The team employed genetic algorithms to refine adversarial samples, iteratively applying mutations and crossovers to maximize evasion success. Results demonstrated that adversarially modified malware files evaded detection in the majority of test cases, validating the effectiveness of gray-box attacks against cloud-based ML defenses.
The success of this attack hinges on the gray-box assumption, where attackers can query the model’s outputs for arbitrary inputs. This scenario is plausible in environments where feature extraction occurs locally, and classification is outsourced to cloud servers. The lack of white-box access—where model internals are exposed—does not preclude effective adversarial manipulation when feature representations are known.
Broader Implications for ML-Based Malware Detection
This research contributes to a growing body of work on adversarial threats to ML-driven cybersecurity. Earlier studies, such as those exploring functionality-preserving PE file modifications or assembler operand changes, demonstrated proof-of-concept attacks but lacked testing against commercial-grade models. The Kaspersky team’s work bridges this gap by validating adversarial techniques on a production-level antimalware system.
- Feature-level exposure: Even without model access, attackers can infer feature representations to craft evasive samples.
- Query-based optimization: The ability to submit arbitrary inputs for classification enables iterative adversarial refinement.
- Functionality preservation challenges: While adversarial modifications must avoid breaking malware, the trade-off between stealth and effectiveness remains a solvable problem for attackers.
The Kaspersky team’s work serves as a case study for the broader AI security community, emphasizing the need for robust adversarial resilience in ML-based threat detection systems.
Sources
- atlas.mitre.orghttps://atlas.mitre.org/studies/AML.CS0014
- securelist.comhttps://securelist.com/how-to-confuse-antimalware-neural-networks-adversarial-attacks-an…
Case study data from MITRE ATLAS, used under the Apache License 2.0.