Additional AI contexts do not boost software warning detection accuracy
Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance
Software EngineeringArtificial Intelligence
Summary
This paper looks at whether adding more language-model-based analysis helps find software bugs better when traditional tools already give clear evidence. The authors tested different setups on software with known warnings, benchmarks, and verification tasks. They found that simply increasing the language model's context size used much more processing but didn’t improve bug clearance. Also, feedback from software verifiers helped somewhat but sometimes increased errors. The results suggest sticking to traditional, deterministic evidence methods works best in these tested cases, while AI methods need refinement.
What this means in practice
- •For software quality teams: Use deterministic tool outputs as primary evidence in software bug clearance instead of relying on expanded AI context counts for improved safety decisions.
- •For software verification engineers: Combine raw verifier feedback cautiously with AI verdicts to improve correct bug detection, while managing increased incorrect verdicts observed in tests.
Authors
S M Abdulla Al Mamun, Akif Islam, Mohd. Ruhul Ameen
Abstract
Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests. On eight related development warnings, increasing the context count from 1 to 50 increases token use 49-fold without improving unsafe clearance. On 160 OWASP cases, model configurations trade clearance against unresolved outcomes; the ten-context arm assesses only 96 cases. On 32 SV-COMP tasks, raw verifier feedback raises mean correct model verdicts from 12.7 to 22.0, but also raises incorrect verdicts from one to seven; a JBMC-plus-JVM validation baseline returns 26 correct verdicts and six unresolved outcomes. On two sets of 120 evidence packages, models reject 70-100% of applicable counterexamples, while construction-specific scope checks classify every package correctly. No target defect is discovered in three Defects4J bugs, one unsupported by the test adapter. These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.