Function level scores measure flag rate more than vulnerability detection quality
A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks
Machine Learning
Summary
Detecting software vulnerabilities using AI language models involves checking if the model flags functions as risky and then clears them after fixes. The authors found that the way these models are evaluated affects the reported scores more than actual model differences. They show many scores mostly reflect how often the model flags code rather than how accurately it detects vulnerabilities. Also, judgments about paired code snippets tend to depend on shared text rather than deeper understanding.
What this means in practice
- •For software security teams: Better interpret vulnerability scan scores by accounting for evaluation protocol effects on flagged function rates in automated tools.
- •For ai model evaluators: Design evaluation tests that separate model impact from protocol influence when assessing vulnerability detection models.
Authors
Maciej Cichoń, Bartłomiej Dmitruk
Abstract
Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured how much of that difference evaluation protocol accounts for, with model outputs held fixed. In a paired test, a model must flag a vulnerable function and clear its version after a fixing commit. Three choices that published evaluations make differently were varied one at a time: metric, verdict extraction and output budget. Seven frontier and large open models were evaluated on five released pair benchmarks and a set pooled for this work under one protocol, and 61 open models of 1.5B to 36B parameters on the pooled set. Function-level F1 follows how often a model flags both functions of a pair (Spearman $+0.86$ over 42 combinations) and is nearly unrelated to pair-level correctness ($+0.16$). On the pair score, extraction changes a model's number by $+0.001$ at the median and budget by $+0.02$ with an interval through zero, whereas the model changes a benchmark's number by up to 0.18 and the benchmark a model's by up to 0.16; a function-level score therefore measures flag rate more than model. For 37 of 68 models the difference between correct and reversed pairs is within its 95% interval of zero, the value for a null model that flags each function at a fixed rate, while both-flagged and both-cleared rates exceed that null by 0.055 on median, and for 64 of 68 both functions of a pair receive one answer more often than independence predicts: verdicts are determined by the text common to both functions. On length-matched pairs a linear probe on activations separates 0.78 by within-pair ranking, against 0.64 for a tf-idf baseline and 0.5 for length; the generated verdict is near chance for three of six models and at 0.55 to 0.57 for the other three, and a prompted logit is at chance for all six.