Compile rate is unreliable metric for fixing code vulnerabilities
Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Software EngineeringArtificial IntelligenceCryptography and Security
Summary
Fixing security problems in C or C++ code using AI models usually checks if the fixed code can compile as a way to tell if the fix works. The authors show that this way of measuring is often misleading because many compile errors come from things outside the AI model’s control and sometimes the 'fixes' aren’t real fixes at all. They suggest using a method that looks only at the actual changes made to the code called diff_F1, which better avoids false credit for no improvement. However, they say even this isn’t perfect and deeper checks that run the code are needed for trustworthy validation.
What this means in practice
- •For software security engineers: Assess AI-generated vulnerability patches more reliably by using change-aware metrics instead of compile rate alone to avoid misleading quality signals.
- •For software testing teams: Use diff_F1 as a fast initial filter to identify meaningful code edits before performing in-depth execution-based vulnerability tests.
Authors
Om Nepal, Sushant Aryal, Oluseyi Olukola, Nick Rahimi
Abstract
Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.