WitnessGym creates automated tests for validating software bugs
WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses
Software EngineeringArtificial Intelligence
Summary
When software has bugs, it's important to show exactly how to trigger them to fix the problem. The authors created WitnessGym, a tool that automatically inserts bugs into real software and generates tests that prove the bugs exist. This makes it easier to evaluate how well coding AI systems can find and demonstrate bugs. They tested several AI coding models and found that producing these bug proofs remains challenging. WitnessGym and its test cases are publicly available for further research and development.
What this means in practice
- •For software developers: Automatically generate tests that demonstrate specific bugs, helping developers confirm and fix faults more reliably.
- •For ai tool builders: Use the benchmark to evaluate and improve AI coding agents’ ability to construct proofs of software bugs.
Authors
Haomin Qi, Xiangzhe Xu, Yiming Huang, Jingbo Shang, Chengpeng Wang
Abstract
Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings actionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present WitnessGym, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifications and execution adapters allow extension to additional bug types and languages. Bug-preserving transformations vary the surrounding structure while preserving the witness behavior. Based on real-world Java projects with test suites, WitnessGym automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available.