Defending and attacking ML Malware Classifiers for Fun and Profit: 2x prize winner at MLSEC-2021

Alejandro Mosquera
4 min readFeb 16, 2023

MLSEC (Machine Learning Security Evasion Competition) is an initiative sponsored by Microsoft and partners CUJO AI, NVIDIA, VMRay, and MRG Effitas with the purpose of raising awareness of the expanding attack surface which is now also affecting AI-powered systems.

In its 3rd edition the competition allowed defenders and attackers to exercise their security and machine learning skills under a plausible threat model: evading antimalware and anti-phishing filters. In the competition, defenders aimed to detect evasive submissions by using machine learning (ML), and attackers attempted to circumvent those detections.

The anti-malware track included two parts: the defensive part focused on creating anti-malware models that were able to withstand adversarial attacks with certain criteria regarding FPR/TPR and an offensive part in which competitors had to bypass the defender models by modifying a set of 50 malware samples provided by the organizers in a way that would still run and produce the same IOCs in an sandboxed environment.

Defensive anti-malware track

PE files have a very large feature space to defend and (even with droppers disallowed this year) it was obvious to me that any malware detection model that I could train with reasonable computing power and public training resources would be relatively easy to bypass.

Taking into account that the feedback loop between submitting a modified malware sample to the MLSEC backend and receiving the execution sandbox results involved a considerable delay (around 1h) and knowing that defeating models in a black box setting would require many queries, I decided to focus on slowing down certain attack paths by aiming for deterrence rather than model robustness. This had the goal of discouraging attackers trying to evade my system with the hypothesis that they will use their resources against easier targets first.

In order to do that I placed several layers of defense in the submitted “A1” solution:

These together did relatively well and placed me in a second “in-the-money” position very close to the winning approach:

There were some other “fun” restrictions, like having to pack the defender as a docker image with certain file size and keeping low latency when analyzing files and providing the verdict. In order to do that the docker base image had to be stripped to the max by deleting certain unused libraries and packages by hand, leaving the OS unstable in some cases…

  • Feature importance of baseline SOREL and EMBER models
  • Lime and SHAP analysis of baseline SOREL and EMBER models:

Offensive anti-malware track

There were 6 defender systems unknown to the participants, however considering the fact that there are not many publicly available large-scale malware datasets besides EMBER and SOREL, It wasn’t unreasonable to believe that attacks that would successful evade baseline models trained on these datasets should have in principle better chances of evasion, or the very least least a better starting point than targeting the defenders directly using black-box attacks.

Therefore the chosen strategy was performing a mix of white-box and black-box attacks against EMBER and SOREL baselines while also extracting knowledge from the evaded defender systems in order to manually fine-tune further attacks in an iterative way.

Other approaches such as PE header fuzzing yielded few but interesting results, such as highlighting the fact that the EMBER feature set makes use of PE header fields ignored by the Windows loader, which opens the door to more interesting adversarial modifications by purposely generating apparently broken PEs which would still run fine in the sandbox. A few more were identified but are left as an exercise to the reader ;).

While droppers were not allowed in this edition, crypters/packers were. I’ve tried a few and had some moderate success for some malware binaries. The main caveat is that for the cases when the evasion fails the packer stub risks being fingerprinted by stateful defenses. Likewise, several malware samples were either loaders themselves or had some sort of integrity checks. These were probably cherry-picked by the organizers to limit packer effectiveness after their success in previous editions.

Overall, generating non-detected variants was relatively easy, the main challenge was generating variants that would evade detection and execute successfully in the organizers sandbox. I chose not testing the modified malware binaries locally before submission, which forced me to use extra API calls (almost 6 times more than the winning team). I was ranked second on this track as well, with just 29 evasions less than the winning team.

Overall, the competition was fun and the experimental results quite interesting, but there are some things the organizers may want to address in future editions. For example, defenders FP rate was enforced only considering clean binaries from a Windows system. This caused some controversy as it was possible to submit a defender that would only detect as clean that particular set and everything else as malware.


amsqr at MLSEC-2021: Thwarting Adversarial Malware Evasion with a Defense-in-Depth

Kipple: Towards accessible, robust malware classification CC10 — Building & Defending a Machine Learning Malware Classifier: Taking 3rd at MLSEC 2021

Originally published at



Alejandro Mosquera

Kaggle Grandmaster. Researcher in AI, Cyber Security, Machine Learning, NLP. Opinions are my own.