Benchmark reveals gap in coding agents finding versus reporting security flaws
VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents
Cryptography and Security
Summary
Some computer programs try to find security problems in software changes but only get scored on their final answer, not how they get there. The authors created VulContextBench, a new test set that checks if these programs look at the right parts of code that actually cause the problem. They found that while programs often look at most relevant code, they only mention a small part when explaining their decision. This shows that current tests miss an important step and programs need to get better at explaining the evidence behind their judgments.
What this means in practice
- •For software security teams: Evaluate and improve AI tools that detect code vulnerabilities by checking if they identify the exact problematic code, not just give correct final answers.
- •For tool developers for coding assistants: Use the benchmark to test and enhance the ability of coding agents to present clear and accurate evidence for their vulnerability findings.
Authors
Yikun Li, Jinfeng Jiang, Yuheng Yieh, Ting Zhang, Yide Yin, Wen Bin Leow, Eng Lieh Ouh, Lwin Khin Shar, David Lo
Abstract
Vulnerability-detection benchmarks score the verdict an agent reaches, not the evidence it gathered. A model that recalls a CVE from pretraining therefore scores the same as one that traced the data flow. We study a task where this difference matters, deciding whether a commit introduces a vulnerability. Instead of scoring the verdict, we score whether the agent retrieved the code its conclusion depends on. We present VulContextBench, a benchmark of 111 vulnerability-introducing commits (VICs) across 83 repositories, 63 CWEs, and five languages. Existing datasets label such commits by tracing a fix back through the version history, which often points to the wrong commit. We therefore audit every case by hand against an explicit four-criterion definition of a VIC, so the benchmark does not inherit that label noise. Each case is annotated with gold context, 464 code blocks in total, each tagged by its role in the evidence for the vulnerability. We evaluate seven frontier models with precision, recall and F1 at three granularities (file, block, and line), scored separately on the context an agent viewed while exploring and on the context it finally declared as evidence. The gap between the two is the main finding. Every model opens most of the gold context while exploring, but reports only part of it as evidence. At the level of code blocks, the share of the gold context a model reports is 37 to 73 percentage points below the share it viewed. Qwen3-Coder-Next views 86.3% of the lines in annotated code blocks but cites only 12.9% in its final report. GPT-5.5, which cites the most, views 73% and reports 36%. These results highlight a gap between finding relevant code and selecting it for the final report, which verdict-level benchmarks cannot reveal.