Agentic software engineering needs standard harnesses to compare tools
Toward Evidence-Based Agentic Software Engineering
Software Engineering
Summary
Success and cost of AI coding helpers depend on both the AI model and the software setup, called the harness, which controls how the model works and interacts. The authors looked at 325 experiments from many papers and found that most did not clearly separate the effects of different harness setups from the AI model itself. They suggest the software engineering community create and use standard harness versions to make experiments more comparable and clear.
What this means in practice
- •For software engineering teams: Use a fixed, community-agreed harness version for fair comparison of AI coding tool features and improvements.
- •For software testing teams: Track and report all software harness settings and versions to isolate causes of test failures in AI-assisted coding workflows.
A position paper. It proposes an approach and reports no results.
Authors
Sebastian Baltes, Christoph Treude
Abstract
Task success rates and execution costs of agentic coding tools depend on both model and harness. The harness manages context, provides tools, and runs the agent loop. In empirical software engineering (SE) studies, comparisons that change model and harness together cannot isolate their effects. We qualitatively annotated 325 experiments reported in 146 peer-reviewed full papers published between 2025 and 2026. We examined the reported configurations and whether comparisons isolated harness effects, including harness configuration and version, model, tasks, and user interaction. The exact version of every harness used was reported for only 15 experiments. For five, the paper reported when the experiments were run, and for 31, authors shared execution traces for every session. Of the 226 experiments comparing harnesses, 35 never varied the harness in isolation. Based on this analysis, we derive requirements for a reference harness that the SE community can use for experiments. We call on the community to report all degrees of freedom of experiments with agentic coding tools and on SE venues to pilot dedicated tracks or special issues around a specified reference harness release. In these tracks, papers would hold the harness version fixed and vary features through separately versioned extensions to isolate feature effects and test their interactions.