The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents
Software EngineeringArtificial Intelligence
Summary
The authors explored how adding different checking tools for AI coding agents affects the quality of software they produce. They tested a simple agent with various tools like a boot probe, shell, and screenshot tool to see if more tools mean better software. Their results showed that basic tools, like a boot probe, mostly help in making sure the app even starts, while more complex tools like screenshots didn’t always improve the software, especially if the problems weren’t visually obvious. Overall, a tool only helps if it can detect the specific way the app fails.
Authors
Achint Mehta
Abstract
Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screenshot tool. We call this set the agent's verification surface. This study asks whether increasing only that surface, with everything else held fixed, produces a matching growth in the quality of the software the agent ships. We built a minimal coding agent whose tool list is the single controlled variable and used it to implement 1,116 web applications across six models and eight tool configurations. A condition-blind human graded every application against a frozen rubric, and automatic probes stress-tested the API-observable behaviors. Verification's cheapest benefit arrives first, which is to make sure that the application comes up. Without any tools, about one build in seven fails to launch at all and a single boot probe removes nearly all of these failures at roughly 35 percent of a full shell's token cost, while the full shell multiplies the no-tools cost by 2.35. Screenshots help most where mistakes are visible (e.g. element placement, interaction), though even there the gain over a shell is modest and does not survive correction for multiple statistical comparisons. In cases where failures can only be measured rather than seen, such as keeping scrolling smooth over a 100,000-row list, screenshots add nothing. A verification tool improves the output artifact only where its reach covers the way the application actually fails.