Coding agents struggle to generate strong test suites from detailed specifications
TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications
Software Engineering
Summary
Creating complete test suites—collections of tests to check if software works as intended—is hard for AI coding agents even when given detailed instructions. The authors built TestSuiteBench, a set of 165 open-source apps with clear behavior rules, code, tests, and variations. They found that these AI agents produce test suites that miss important cases and don't catch many hidden bugs, although the generated code itself passes most reference tests. This shows that while AI can write software, verifying it thoroughly is still a challenge.
What this means in practice
- •For software testing teams: Evaluate and improve test generation tools by comparing against TestSuiteBench’s comprehensive test suite benchmark for real-world applications.
- •For software development teams: Understand limitations of AI-generated tests to better supplement automated testing with manual reviews and additional coverage.
Authors
Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller, Martin Vechev
Abstract
Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73\% are killed by exactly one reference test and, on average, by only 0.2\% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60\%, despite their suites being largely valid and 88\% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82\% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.