Quantum model detects malware images but faces smart attack challenges
Q-MALDetect: Quantum Image-Based Malware Detection and Adversarial Robustness Analysis
Cryptography and Security
Summary
Malware detection tries to find harmful software before it causes damage. This paper introduces a new approach using quantum machine learning to analyze images created from computer program files. While the model can identify malware well in normal cases, it becomes vulnerable when tricky attackers make small changes to fool it. The researchers tested several attack methods and defenses, finding that some defenses work better than others but no solution is perfect yet. This shows tools like this need thorough testing against different attacks before being used for real security.
What this means in practice
- •For cybersecurity engineers: Improve malware scanning systems by evaluating quantum machine-learning models under sophisticated code-aware evasion techniques.
- •For quantum software developers: Design quantum classifiers for security tasks with attention to adversarial robustness and feature encoding strategies.
Authors
Walid El Maouaki, Mohammad Izen, Nouhaila Innan, Alberto Marchisio, Muhammad Shafique
Abstract
Quantum machine-learning (QML) models are increasingly considered for security applications, yet their adversarial robustness in malware analysis remains insufficiently understood. This paper presents Q-MALDetect, a hybrid convolutional and variational-quantum detector for byte-image representations of x86-64 Executable and Linkable Format (ELF) binaries, and evaluates it under feature-space and a range of Grad-CAM-guided, executable-aware evasion attacks. The model achieves 97.2% training accuracy, but its robustness deteriorates as attack strength increases, with the strongest evaluated feature-space attack reaching a 78.7% attack success rate (ASR). Among executable-aware attacks, Grad-CAM-guided no-operation (NOP) insertion produces the highest conditional ASR of 32.9%. Dynamic adversarial training is the most effective evaluated defense, reducing feature-space ASR to as low as 5.0%, executable-aware ASR to 7.7%, and the sequential-attack ASR to 34.0%. In contrast, convolutional-autoencoder preprocessing increases executable-aware ASR to 39.7%, coinciding with a rise in Grad-CAM attribution mass within the mutable .text region from 35.2% to 45.7%. Ablation results reveal no monotonic relationship between qubit count and robustness, while the encoding interface modifies the learned attack surface without neutralizing the strongest attack. These findings demonstrate that the effectiveness of attacks and defenses depends strongly on the perturbation constraints and evaluation conditions. Reliable deployment of QML malware detectors therefore requires assessment against both feature-space and executable-aware attacks rather than reliance on clean accuracy or generic denoising alone.