Direct malware classification outperforms claim scoring in language models
MARS: Malware Analysis with Rule-Based Scoring of LLM Claims
Cryptography and SecurityArtificial Intelligence
Summary
Malware is harmful software that people want to detect automatically. The authors compare two ways large language models can decide if a file is malware: giving a direct yes/no decision or making several smaller behavioral claims that get scored by rules. They find that having the models give a direct malware verdict is usually more accurate than scoring claims. However, keeping the claims helps understand and change rules later without running the model again.
What this means in practice
- •For security operations teams: Improve malware detection accuracy by using direct verdict outputs from language models rather than relying on scored behavioral claims.
- •For security software developers: Incorporate claim retention to enable post-analysis policy updates without rerunning large language models.
Authors
Hyeongjun Choi
Abstract
Large language models can triage malware through direct verdicts or behavioral claims scored by an external policy. We present MARS, a malware triage framework, and compare direct classification with single-pass claim scoring using the same evidence collector and identical static evidence bundles for each model. The evaluation covers 1,195 PE and ELF binaries grouped into 1,001 near-duplicate clusters and six language models, with deterministic rules providing a baseline. Direct classification is more accurate for all six models. On samples with usable outputs from both paths, its accuracy advantage ranges from 3.7 to 20.9 percentage points, with all 95% cluster-bootstrap confidence intervals for the differences above zero. It also achieves higher malicious alert recall in ten of twelve platform and model combinations. Claim mediation provides no consistent reduction in performance variation across models. Separate subset studies find more consistent alert decisions for direct classification and a larger recall loss for the claim path when predefined indicator fields are removed. In a family identification probe, claims yield higher accuracy than verdict labels but lower accuracy than evidence text. Retained claims expose the inputs to verdict computation and permit policy revision without another model call. We reproduce archived verdicts exactly and apply a revised policy to the same records, including outputs from two additional models withdrawn by their provider. Under the evaluated claim taxonomy and additive policy, these results favor direct classification when only a verdict is required, while demonstrating that retained claims support explicit policy inspection and revision.