AI and static analysis tools find different security bugs in code
How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
Cryptography and SecurityArtificial IntelligenceSoftware Engineering
Summary
Finding dangerous bugs in software is tricky because different tools spot different problems. The authors tested AI coding helpers alongside a standard bug-finder called static application security testing (SAST). They found that the AI tools and SAST each catch many unique bugs the other misses, so using both together finds the most vulnerabilities. The way AI is set up and used can affect its success, not just the AI itself. They made all their test cases and results publicly available so others can build on their work.
What this means in practice
- •For software security teams: Improve vulnerability detection by combining AI coding agents with static analysis to cover more security flaws in web applications.
- •For devops engineers: Tune AI security reviews and static analysis settings to optimize coverage and cost trade-offs when scanning multi-language codebases.
Authors
Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark
Abstract
We ask what AI coding agents and AI security harnesses find that deterministic static application security testing (SAST) does not, and what they miss that SAST catches. VulnBench 2.0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java; 62 flow across more than one file. A finding is credited only if it locates the vulnerability's attacker-controlled source and its dangerous sink, each within two lines. We evaluate nine productized configurations under one blind protocol: the Snyk Code static analyzer, Claude Code's /security-review with three model and effort settings (one with the Snyk MCP server), Codex Security with GPT-5.6 Luna, Terra and Sol, and the DeepSec scanner with two model backends. Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they disagree on 28 vulnerabilities, split 16 to 12, too few to tell them apart. Neither engine was sufficient alone: together they reached 101 of 108, the ceiling of all seven comparable configurations combined, and the AI configurations also reported at least seven real vulnerabilities that the reference set did not contain and Snyk Code did not report. What an AI engine misses depends on its harness as much as on its model: without Claude Code's built-in review policy, the same model matched 16 instead of 9 of the 20 vulnerabilities in the classes that policy excludes or restricts (supplementary run on 19 of 20 applications); Codex Security with GPT-5.6 Sol located 80 vulnerabilities at file level but 34 at line level; DeepSec's model-free file selection kept a file from only 56 of the 108 flows. Coverage and cost were only loosely related. The harness, fixtures, reference sets and all run data are public.