VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
2026-08-12 • Cryptography and Security
Cryptography and SecurityArtificial IntelligenceComputation and LanguageSoftware Engineering
AI summaryⓘ
The authors created a new benchmark dataset called VICBench to help test tools that find security problems in code. VICBench includes 100 carefully checked commits that introduced vulnerabilities in three programming languages across many projects. These commits are bigger and more complex than those in earlier datasets. When the authors tested current detection algorithms on VICBench, the results showed these tools still miss many vulnerabilities, meaning manual review is needed. VICBench provides a better way to evaluate and improve vulnerability detection methods.
vulnerability-inducing commitsCVECWEbenchmark datasetsecurity vulnerability detectionV-SZZLLM4SZZprogramming languagessoftware patches
Authors
Jin Lu, Xuening Han, Yang Zhong, Lin Tan, Kevin Luo, Andrew Gacek, Neha Rungta
Abstract
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.