Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits
2026-08-10 • Software Engineering
Software Engineering
AI summaryⓘ
The authors studied how large language models generate secure code and how tests can help fix vulnerabilities. They created SecTDD, a system that uses tests shown before code generation and during repairs to improve security. Their experiments showed that showing all tests upfront generally helps, but not always, and that detailed feedback can fix many issues without breaking working parts. However, even when code passed visible tests, it sometimes still failed hidden checks, highlighting the importance of thorough test coverage. Overall, the study shows test feedback can improve secure code generation but results vary depending on several factors.
large language modelssecure code generationexecutable specificationstest-driven developmentsecurity vulnerabilitiescode repairsoftware testingCWE categoriesfeedback mechanismsbenchmark evaluation
Authors
Yunhao Liang, Chengguang Gan, Ruixuan Ying
Abstract
Large language models (LLMs) can generate functionally useful code that remains vulnerable, while security-focused interventions may break intended behavior. We investigate security tests as executable specifications both before generation and during iterative repair. We develop SecTDD, a controlled test-feedback scaffold that separates three factors: whether tests are shown upfront, whether failed executions trigger revision, and how failures are selected and represented. The evaluation uses behavior-partitioned visible and hidden tests and byte-identical initial candidates for repair comparisons. Across 2,705 trajectories, 31 task instances, three secure-code benchmarks, 16 CWE categories, and two model families, showing all visible tests upfront increases hidden functional-and-security joint success by 19.3 percentage points on average, but improves only seven of nine benchmark-model conditions and harms two. In shared-candidate comparisons, structured feedback repairs 80 initially unsuccessful candidates with no joint regressions; fixed raw feedback repairs 83 but causes three regressions. Structured and raw feedback are otherwise nearly indistinguishable head-to-head (six wins, six losses, and 453 ties). Candidates that pass all visible tests still fail hidden behavior families under every common regime. These results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage.