Case level verification improves false positive reduction in security testing
Case-Level Verification in Scanner-LLM Cascades: Overcoming the Alert Aggregation Bottleneck to Expand the FRR-TPR Trade-off Space
Cryptography and Security
Summary
Security scanners used to find vulnerabilities often report too many false alarms, which wastes time checking them. Large language models (LLMs) can detect almost all real problems but also create many false alarms, so they can’t replace scanners on their own. The authors found that grouping alerts together creates a problem because fixing one false alarm may hide true vulnerabilities. They propose verifying each alert case individually using LLMs, which lets them better separate false alarms from real threats. Their approach improves the balance between catching real issues and reducing false alerts.
What this means in practice
- •For cybersecurity teams: Enhance vulnerability triage by verifying each alert individually to reduce false alarms while retaining true positives in web application testing.
- •For web application developers: Improve automated security testing workflows by integrating per-case verification to better prioritize real issues over false alerts.
Authors
Hao Sun, Yibin Yao, Chaohai Xie, Yuqun Lin
Abstract
Dynamic Application Security Testing (DAST) scanners achieve high recall but also produce a large number of false positives, resulting in substantial manual triage costs. Large Language Models (LLMs), when used for independent detection, achieve extremely high recall (95.4%-100%) but also exhibit prohibitively high false positive rates (49.6%-85.0%), precluding their use as standalone replacements for scanners. A natural solution is a two-stage cascade consisting of scanner detection followed by LLM verification. However, a verification-granularity issue that has long been overlooked in practice creates a structural bottleneck: alert aggregation binds multiple true and false cases into a shared decision unit, such that removing a false positive inevitably eliminates true positives aggregated within the same alert group. This creates a trade-off bottleneck between the False-positive Reduction Rate (FRR) and the True-positive Rate (TPR). We formalize this bottleneck by showing that the alert-level false-positive set is a subset of the case-level false-positive set, and introduce a Case-Level, per-case verification strategy that shifts the decision granularity from the alert level to the instance level, independently replaying HTTP requests and making an independent determination for each detected case. Evaluation on the dual testbeds of Damn Vulnerable Web Application (DVWA) and WebGoat shows that the empirically best Alert-Level operating point achieves FRR=42.86% (TPR=51.7%). The Case-Level Baseline achieves FRR=47.6%, an improvement of 4.7 percentage points (+4.7 pp), while the Case-Focused Evidence Verification Prompt (CEV-Prompt) increases TPR from 55.2% to 62.1% at the same FRR.