Agent software harnesses have many untested bugs found by new tool
Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
Software EngineeringArtificial Intelligence
Summary
Complex software that runs AI agents often has hidden bugs because parts of their supporting programs don't get tested enough. The authors studied these agent harnesses and found that key code related to AI outputs is under-tested. They created a tool called HarnessTester to better test these programs. This tool finds many more bugs than previous methods, helping improve agent software reliability.
What this means in practice
- •For software engineers: Improve testing practices for AI agent software harnesses to find hidden bugs and increase reliability.
- •For quality assurance teams: Use HarnessTester to automatically generate tests that better cover critical AI harness code and detect real bugs.
Authors
Yifan Xiong, Jingyi Ge, Zhenpeng Chen, Yiling Lou
Abstract
LLM-based agentic systems are emerging as a new software paradigm. Modern agents are typically composed of backbone LLMs and a surrounding harness that serves as the operational software infrastructure for agent execution. As agent harnesses grow increasingly complex, agents suffer from diverse harness implementation bugs, raising substantial reliability concerns. In this work, we conduct the first empirical study to systematically investigate the test adequacy of harness in real-world agentic systems. Our analysis reveals that agent harness remains substantially undertested. In particular, LLM-dependent harness (LDH) code, despite its critical role in processing LLM outputs and governing agent behavior, receives limited testing attention, with less than half of its lines and branches covered by existing tests. Motivated by these findings, we further propose HarnessTester, the first harness-oriented test generation technique that incorporates explicit agent-harness contract support to construct contract-faithful test setups and extensively exercise LDH code. Our evaluation shows that HarnessTester substantially outperforms state-of-the-art general-purpose test generation techniques in achieving 75.95%/84.76% larger line/branch coverage gains and 69.89% larger mutation-score gains. Furthermore, HarnessTester detects 122 real-world harness bugs in widely-used agentic systems (e.g., OpenClaw), among which, 88 bugs are previously-unknown bugs and 69 bugs have been confirmed by agent developers. These results highlight the practical effectiveness of HarnessTester in improving test adequacy and assuring the reliability of real-world agentic systems.