Malware image benchmarks overstate accuracy due to duplication and leakage
Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks
Cryptography and Security
Summary
Malware files are sometimes turned into pictures so computer programs can identify them more easily. This study found that many test sets used to check these programs have many duplicates and overlaps between training and testing data. Because of this, the programs appear much more accurate than they really are. When duplicates are removed carefully, accuracy scores drop significantly. The researchers recommend checking for duplicates and comparing image-based methods with other non-image approaches.
What this means in practice
- •For malware analysts: Improve evaluation of malware classifiers by detecting duplication and leakage in image-based datasets to avoid overestimated accuracy.
- •For cybersecurity engineers: Use findings to validate malware detection systems better by including non-image baselines and controlling for near-duplicate samples in training data.
Authors
Stefan Larson, Kevin Leach
Abstract
Classifying malware binaries rendered as images is a popular approach to malware family identification, and papers on the standard benchmarks for this task report accuracies of 97-99%. In this paper we audit ten of these benchmarks for near-duplicate images and test-train leakage, and retrain standard classifiers under naive and content-aware splits. Deduplicating by file hash, which is the standard practice, does not prevent leakage, because distinct binaries render to identical images. Deduplicating on image content reduces top-1 accuracy by up to 8 points and macro-F1 by up to 15 points on the released benchmarks we find leaky. The rendering step also adds duplication, since byteplots of the same binaries contain 19.7% near-duplicates where static feature vectors contain 2.2%, and changing the width at which bytes are wrapped into rows reduces accuracy by 20 to 33 points, although no byte changes. We recommend that content-level duplication and a non-image baseline be reported alongside accuracy.