Papers for

automation tool developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Coding agents waste money with inefficient behaviors and improvements vary

Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agents

Abstract: Although effective, coding agents often incur substantial monetary costs. Their recurring cost-inefficient behaviors remain underexplored. We conduct the first study of behavioral cost inefficiencies in coding agents, analyzing 1,200 trajectories from Claude Code and Mini-SWE-Agent across four configurations on SWE-bench Verified. We identify three cost-inefficient behaviors: subsumed retrieval, similar script generation, and test re-execution. We then evaluate three mitigation strategies: structure-aware retrieval, agent-synthesized skills, and developer-designed skills, over 10k trajectories on held-out SWE-bench Verified and Pro tasks. Our main findings are: (1) The three behaviors affect 79.00\%--98.00\% of coding tasks and account for up to 22.75\% of task cost. (2) Structure-aware retrieval can introduce retrieval overhead and alter agent delegation, causing inconsistent improvements in retrieval efficiency and cost increases of up to 28.14\%. (3) Agent-synthesized skills tend to produce low-level, trace-specific guidance, limiting their effectiveness and generality. (4) In contrast, developer-designed skills provide high-level, trace-agnostic guidance, reducing cost by up to 41.73\%, roughly twice the maximum gain from agent-synthesized skills.

Fri 25 SeptArtificial IntelligenceSoftware Engineering
The gist
Coding agents help write software but often waste money in how they work. The authors studied behaviors that cause this waste, like repeating searches or rerunning tests unnecessarily. They tested ways to fix these problems, finding that developer-designed skills worked best to save costs. Agent-made skills and smarter retrieval didn’t always help and sometimes made things worse. This shows better ways to make coding agents cheaper and more efficient.
Open → 2609.30725v1

LLM choice impacts Python test reliability more than prompt style

Evaluating the effectiveness of class-level LLM-generated test suites in Python

Abstract: Context: Large language models (LLMs) can generate unit tests quickly, but high structural coverage does not establish that those tests execute reliably or detect faults. Existing evidence often treats coverage as the principal outcome and rarely compares prompt strategies and models through mutation testing at class level. Objective: This study examines how prompt strategy and model choice shape the executability, structural coverage, fault-detection effectiveness, and structural quality of LLM-generated Python test suites relative to human-written suites. Method: We evaluate multiple prompt strategies across a diverse set of current LLM configurations on the ClassEval benchmark. The evaluation combines execution outcomes, line and branch coverage, Cosmic Ray mutation scores, and structural quality indicators. Primary analyses treat successful execution as a prerequisite; paired comparisons use only classes shared by the relevant executable subsets. Results: Structural coverage is consistently near its ceiling and offers little discrimination among configurations. Executability varies substantially. The proposed prompt performs strongly for mutation score, but no prompt dominates across models. Model choice explains more variation than prompt choice, and their interaction shows that prompt effectiveness depends on the selected model. Human and LLM suites are evaluated on unequal executable subsets, so their relative mutation scores do not establish superiority. Conclusion: Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators. In practice, model selection should precede prompt tuning.

Mon 21 SeptSoftware Engineering
The gist
Generating Python unit tests with language models often produces many tests, but having lots of test coverage does not guarantee the tests run well or catch bugs. The authors studied how different ways of prompting and which LLMs are used affect how reliably the tests run, how much code they cover, and how good they are at spotting faults. They found the model used matters more than the prompt style and that test coverage alone isn’t enough to judge quality. They recommend making sure tests can run before checking their quality and using mutation tests together with coverage metrics.
Open → 2609.24341v1