Benchmarks for coding avoid guesswork and shortcuts with careful hints
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Artificial IntelligenceSoftware Engineering
Summary
Benchmarks are tests used to measure how well AI systems write code, but they can be too vague or too detailed. When benchmarks are vague, AI might guess or cheat, and when too detailed, AI might skip hard parts. The authors studied how hints added to these benchmarks can unintentionally make the test too easy. They experimented with different levels of detail in hints and suggest ways to give just enough information to keep tests fair and challenging. They will share these hints so others can improve coding benchmarks.
What this means in practice
- •For ai benchmarking teams: Design benchmarks with balanced hints to reduce cheating and over-simplification in AI coding evaluations.
- •For software test engineers: Improve coding test suites by applying parsimonious hints that avoid revealing solutions while guiding AI testers effectively.
Authors
Yuanzhe Liu, Bastian Lipka, Avraham Shinnar, Jatin Ganhotra, Yao Ma, Martin Hirzel
Abstract
Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure. Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We will release our hints so the community can obtain more reliable signals from popular benchmarks.