VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection
2026-08-03 • Software Engineering
Software Engineering
AI summaryⓘ
The authors created VulnGym, a new benchmark to test how well coding agents find security problems in entire code repositories, not just small code snippets. VulnGym links real GitHub security advisories to the exact code versions that contain vulnerabilities, providing detailed annotations about where and how the problems occur. This helps researchers evaluate both the detection of vulnerabilities and the understanding of the underlying code issues. Their tests show that current coding agents still struggle with fully detecting vulnerabilities and explaining their findings in large code bases.
LLMcoding agentsvulnerability detectionbenchmarksGitHub advisoriesrepository-level analysiscode localizationvulnerability tracessecurity vulnerabilitiessoftware security
Authors
Kexing Ji, Jiachen Liu, Enze Hu, Cuiyun Gao, Keke Lian, Yongheng Liu, Lei Zhang, Tian Dong, Hao Chen, Wang Bin
Abstract
Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present \textbf{VulnGym}, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repository-level vulnerability detection and the construction of accurate supporting traces.