EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability
2026-07-27 • Cryptography and Security
Cryptography and SecurityMachine Learning
AI summaryⓘ
The authors created EXE-Bench, a thorough test system for AI tools that detect Windows malware, because previous tests were inconsistent and incomplete. Their benchmark checks not just accuracy but also whether these tools stay effective over time, resist hacking attempts, and run efficiently on computers. They found that tools relying on expert knowledge to pick features work better over time and against attacks than most deep learning models, which perform well only right after being trained. This shows the importance of careful testing before deploying malware detectors in real-world use.
Windows malware detectionAI-based malware detectorfeature engineeringtemporal robustnessadversarial attacksbenchmarkingdeep learningcomputational overheadmodel evaluation
Authors
Andrea Ponte, Daniel Gibert, Matous Kozak, Dmitrijs Trizna, Maura Pintor, Battista Biggio, Fabio Roli, Luca Demetrio
Abstract
Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.