Agent supported pull requests change tests differently than humans do
Testing with Coding Agents: A Large-Scale Comparison of Agent-Supported and Human-Authored Pull Requests
Software Engineering
Summary
Developers use software tools called coding agents to help write tests in their code. This study compares test-related changes in code pull requests made by humans alone versus those with coding agents helping. The authors found that agent-supported changes add tests more often but modify or remove existing tests less. The overall quality and coverage of tests appeared similar, but agent-made tests sometimes cover less code in test-only updates. The long-term impact of using agents on test quality remains unclear.
What this means in practice
- •For software development teams: Assess and balance the division of test maintenance work between developers and coding agents in pull requests to improve long-term codebase health.
- •For software quality assurance teams: Monitor differences in test quality and coverage when coding agents contribute tests via pull requests to better guide test review and acceptance decisions.
Authors
Roberto Milanese, Francesco Salzano, Hovhannes Muradyan, Jingjing Wu, Remo Pareschi, Fausto Fasano, Mattia Fazzini
Abstract
Context. Coding agents increasingly contribute tests through pull requests that developers then maintain, so how testing work is divided affects long-term maintainability. Objective. We compare pull requests in which developers and coding agents collaborate (agent-supported) with pull requests without detectable agent involvement (human-authored) along five dimensions: the frequency and extent of test changes, their type and context, test structure, test smells, and code coverage. Method. We analyze 114,134 pull requests merged between January 2025 and April 2026 in 391 Java repositories, identifying five coding agents through pull-request-level and commit-level signals. Since agent adoption varies across projects, we compare the two groups within repositories with mixed-effects models that control for change size. Results. Pooled across repositories, both groups change test code almost equally often; within repositories, the odds that an agent-supported pull request does so are 1.65 times higher. Agent-supported pull requests add tests more often (odds ratios 1.50-1.58) and modify (0.70-0.76) and delete (0.51-0.77) them less often. We detect no difference in most structural characteristics of the tests they add, which more often have an oracle, nor, in the 95 repositories we could build, in the coverage of their production changes; in test-only pull requests, however, their tests cover less of the code they exercise. Differences in test smells are borderline and not reproduced for individual test methods. Conclusions. Agent participation is associated with a different mix of testing work, which pooled comparisons across projects can hide or reverse; its relationship with test quality remains uncertain.