Language model systems struggle to find vulnerable code in big software projects
Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
Cryptography and SecurityArtificial Intelligence
Summary
Finding exactly which files in a large software project are vulnerable to security issues is hard. The authors created a new benchmark called VLoc Bench using 500 real security problems in popular software repositories. They tested 27 language models and several code analysis tools on how well they could pinpoint the exact files with vulnerabilities. The best system could only correctly identify vulnerable files about 23% of the time, and many tasks were missed completely. This shows that locating vulnerable code is a difficult but important capability that is different from just detecting or fixing problems.
What this means in practice
- •For software security teams: Identify which files in large unfamiliar codebases are vulnerable based on known weakness types to prioritize inspection efforts.
- •For automated security tool developers: Develop more reliable vulnerability localization agents that can avoid false alarms when vulnerabilities are already fixed.
Authors
Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, Amin Karbasi
Abstract
Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code. We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness. We introduce the Vulnerability Localization Benchmark (VLoc Bench), comprising 500 real world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix. On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access and must return the affected files; on the patched snapshot, it must determine that the recorded vulnerability is no longer present. We evaluate 27 language models and four static-analysis tools under a common agent interface. Repository-scale vulnerability localization remains difficult: the strongest system achieves 0.229 File F1, and 38.4% of tasks receive no correct localization from any evaluated model. We further find that stronger localization does not imply reliable behavior after remediation: systems that identify vulnerable files effectively can still report unsupported locations on patched repositories. These results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.