RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities
2026-07-06 • Cryptography and Security
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
AI summaryⓘ
The authors created RustMizan, a new tool to test how well large language models (LLMs) can find security problems in Rust code. Existing tests use small code pieces and simple yes/no results, but RustMizan uses real, working Rust code with more detailed labels showing where issues are. They also made a way to tweak code without changing its meaning to check if the models get confused. When testing four top models, the authors found that the models can only tell if code is vulnerable about 56-65% of the time, and are much worse at pinpointing exact problem lines.
Large Language ModelsRust Programming LanguageVulnerability AnalysisBinary ClassificationCode LocalizationCommon Weakness EnumerationSemantics-Preserving MutationsBenchmarkingAdversarial Testing
Authors
Tarek Elsayed, Shiping Yang, Eunsong Koh, Sanika Goyal, Vincent Huang, Paul Ngo, Nathan Young, Mohammad Omidvar Tehrani, Alvyn Kang, Arnell Kang, Zeyu Chen, Angélica Moreira, Xuan Feng, Angel X. Chang, Nick Sumner, Steven Y. Ko
Abstract
LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace. They typically rely on small non-compilable snippets, focus on binary classification (vulnerable or not), and do not account for the risk that publicly-released datasets are part of model training corpora. We introduce RustMizan, a benchmarking framework for Rust vulnerability analysis that addresses these gaps. RustMizan contains compilable code variants at the crate, file, and function levels, with annotations for binary vulnerability detection, CWE classification, and function- and line-level localization. A paired mutation framework produces semantics-preserving code mutants for contamination testing and robustness probing. Across four frontier models in an agentic setup with command-line access, binary classification sits in the 56-65% range, but line localization F1 stays near 20%, and adversarial cues drop line F1 by about 27%.