Malware evasion is improved using adversarial autoencoder with double critics
Mal-AAEGAN: Adversarial Autoencoder with Double-Critic WGAN-GP for Malware Generation
Cryptography and Security
Summary
Machine-learning tools that detect malware can be tricked by slightly changing malware samples, but traditional tricks struggle because malware features are complex and binary. The authors propose Mal-AAEGAN, which learns a simpler, continuous representation of malware features to create effective hidden changes. They use two 'critics' to guide the process, helping generate diverse adversarial malware that still behaves like the original. Their approach is tested on two malware datasets and beats or matches prior methods in evading eight common detectors.
What this means in practice
- •For security software teams: Develop improved tools to test and strengthen malware detectors against evasive attacks using stable adversarial malware generation in continuous feature space.
- •For cyber threat analysts: Use advanced adversarial malware samples generated by Mal-AAEGAN to better understand and anticipate attacker evasion strategies.
Authors
Nam Hoang Thanh, Trung Pham Duy, Lam Bui Thu
Abstract
Machine-learning (ML) malware detectors are known to be vulnerable to adversarial samples. Generative adversarial networks (GANs) such as MalGAN, Mal-LSGAN, and our previous Mal-D2GAN can craft adversarial malware to probe and harden these detectors, yet they operate directly on a high-dimensional, discrete binary feature space, which leads to unstable training, vanishing gradients, and adversarial samples of limited diversity. To address these problems we propose Mal-AAEGAN. First, an adversarial autoencoder (AAE) learns a low-dimensional, regularized and continuous latent representation of the $M$-dimensional binary API feature vector, and the generator performs an additive perturbation in this latent space rather than in the raw binary space. Second, training follows the Wasserstein GAN with gradient penalty (WGAN-GP) scheme using two critics: a mimic-black-box critic that inherits the role of the substitute detector and a naive critic that inherits the role of the additional detector. A decode-binarize-OR pipeline guarantees that the original API calls of the malware are preserved. The model is evaluated on two malware datasets of different input dimensionality ($M\in\{139,215\}$) against eight common ML detectors and compared, under an identical protocol, with MalGAN, Mal-LSGAN and Mal-D2GAN. Mal-AAEGAN trains stably in the continuous latent space and attains an evasion rate on par with or higher than these binary-space baselines across both datasets, while its generator stays responsive to the noise input rather than reducing to a single fixed perturbation per malware.