Vulvalidate improves accuracy of function vulnerability labels with runtime tests
VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
Summary
Many datasets that flag pieces of software as vulnerable do so only because those parts were changed during a security fix, which can be misleading. The authors created VulValidate, a system that runs actual tests on software to confirm if a function was truly vulnerable. By applying VulValidate to large vulnerability datasets, they found many labels were wrong or uncertain and produced corrected and verified labels. This helps make vulnerability detection tools more reliable and trustworthy.
What this means in practice
- •For security software developers: Use VulValidate to audit and improve the accuracy of vulnerability labels in training datasets, enhancing the performance of security detection tools.
- •For software testing teams: Employ VulValidate to create executable tests that confirm if functions are truly vulnerable, reducing false positives in security assessments.
Authors
Leizhen Zhang, Sheng Chen
Abstract
Reliable learning-based vulnerability detection requires high-quality labels, yet datasets built from vulnerability-fixing commits may label functions as vulnerable simply because they were changed by a security patch. We present VulValidate, a framework that uses LLM agents to coordinate dynamic analysis tools and construct vulnerability-triggering experiments from runtime feedback. Given a labeled function and its fixing patch, VulValidate reconstructs vulnerable and fixed revisions, selects suitable tools and execution paths, refines triggering inputs, and compares runtime behavior to assess function-level attribution. We audit all 35,849 instances originally labeled vulnerable in BigVul, PrimeVul, and DiverseVul. We confirm 20,510 (57.2%), correct 6,819 labels (19.0%), leave 7,981 attacked but undecided (22.3%), and cannot successfully measure 539 (1.5%). After conflict resolution and byte-exact deduplication, the corrected release contains 15,890 distinct confirmed vulnerable function bodies. In a blinded review of 581 sampled decisions, expert consensus supports 90.0%--92.0% of confirmations and 92.6%--99.0% of label corrections. With model parameters fixed, corrected evaluation lowers F1 for all five tested detectors on both BigVul and DiverseVul; retraining with corrected labels improves F1 for four of five detectors on each dataset. We also release a reusable VulValidate skill, corrected datasets, and reproducible evidence for future vulnerability-detection research.