Summary
Benchmarks for testing automated program repair tools have usually been fixed collections of real defects, which limits how much you can control experiments and reuse benchmarks safely. The authors propose making benchmarks from detailed, executable specifications that describe exactly what kinds of bugs, difficulty, and validation methods should be included. This lets benchmarks be generated on demand, verified independently, and adapted to new testing needs instead of relying on old, unchanging datasets. Their approach turns benchmark creation into a clear, repeatable design process rather than just curating examples.
What this means in practice
- •For software testing teams: Generate tailored program repair benchmarks with controllable properties to better evaluate automated repair tools against diverse and verifiable defect scenarios.
- •For program repair tool developers: Build benchmarks based on explicit specifications to systematically test and improve their tools on definable fault types and difficulties instead of static datasets.
A position paper. It proposes an approach and reports no results.
Abstract
Automated Program Repair (APR) benchmarks have traditionally been constructed as static datasets whose characteristics are inherited from the defects they contain. While this paradigm has enabled decades of progress, finite corpora provide limited experimental control, become increasingly susceptible to contamination as they are reused, and cannot be systematically regenerated or adapted as evaluation requirements evolve. We propose specification-driven benchmarking, a paradigm in which benchmarks are defined by executable specifications and realized through benchmark generation. The specification explicitly declares the intended properties of the benchmark (including program context, fault taxonomy, difficulty, validation strategy, and corpus constraints) while a generation pipeline realizes those requirements through independent generation, validation, and corpus management components. We develop the conceptual foundations of this approach by introducing a taxonomy of benchmark specification dimensions, establishing how each specification dimension maps to deterministic architectural responsibilities, and arguing that independent validation is a structural requirement for trustworthy benchmark generation. An end-to-end example illustrates how specification choices propagate through the pipeline to produce benchmark instances whose properties are independently verifiable. By treating the benchmark as an executable specification rather than a static dataset, the proposed paradigm shifts benchmark construction from artifact curation to declarative experimental design.