Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models
2026-06-03 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors studied a method called Vul-RAG that uses large language models to find software bugs. They checked if the method's results can be repeated using open and local versions of these models. Their tests showed similar results but found that making the models bigger or newer doesn’t really improve bug-finding performance beyond a certain point. They also looked at how different types of models affect results and shared their code for others to use. This helps understand what really matters for detecting vulnerabilities with AI.
Large Language ModelsSoftware Vulnerability DetectionRetrieval-Augmented GenerationReproducibilityOpen-weight ModelsCode-specialized ModelsPairwise AccuracyModel CapacitySource Code Analysis
Authors
Sabrina Kaniewski, Fabian Schmidt, Tobias Heer
Abstract
Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings. However, for approaches relying on proprietary models and APIs, reproducibility and replicability remain largely unexplored, raising the question of whether reported results generalize or depend primarily on specific model choices. In this work, we present a reproducibility study of Vul-RAG, a RAG-based framework for source code vulnerability detection that enhances LLMs with high-level vulnerability knowledge. We first replicate the results in a fully local and open-weights setting using the reported open-weight baseline models. We then extend the evaluation to a diverse set of recent open-weight LLMs, including code-specialized, general-purpose, and reasoning models of varying parameter sizes. The results confirm that the findings of Vul-RAG are reproducible under local deployment, but with minor deviations. Across all evaluated models, we observe a performance plateau at approximately 0.30 pairwise accuracy (code pairs for which both the vulnerable and the patched function are correctly classified). Notably, this plateau persists even for more recent and advanced models, indicating that improvements in model capacity alone do not substantially enhance performance. Finally, we discuss practical implications and trade-offs between detection effectiveness, model capabilities, and model scale. Implementation and evaluation artifacts are publicly available at https://github.com/hs-esslingen-it-security/revisiting-Vul-RAG.