Small test suites improve self-evolving software agents’ rewrite checks
Small but Mighty: Test Suites as Improvement Signals for Self-Evolving Agents
Artificial IntelligenceSoftware Engineering
Summary
When software agents rewrite their own code, they need to check if the changes are good or bad. The authors show that existing test suites used to check these rewrites often miss many important mistakes. They designed a way to find small but strong sets of tests that catch more problems without being costly to run. These improved test suites help the self-evolving agents better decide which rewrites to keep.
What this means in practice
- •For software development teams: Improve automated code rewriting systems by using small, strong test suites that better detect problematic changes.
- •For quality assurance teams: Use compact test sets derived by evolutionary search to efficiently validate behavior changes in evolving software.
Authors
Yutong Cheng, Wei Cheng, Wenchao Yu, Dawei Zhou, Haifeng Chen
Abstract
A self-evolving agent rewrites its own harness and must decide which rewrites to keep, so its improvement signal is the check a rewrite has to pass. Verifying a rewrite by agent rollout is expensive and subjective, while executable checks such as the repository's own tests are too sparse to catch behavior-changing rewrites, rejecting only a quarter of them. We instead take test feedback as the executable signal, and search for test suites that are both small and strong with a quality-diversity evolutionary algorithm, so that the signal is dense and informative. TeSearch treats suite size as a behavior descriptor, so that quality is pursued at every size and a small, strong suite is reachable, and learns a transition kernel during the search that denoises the LLM as an actuator on that descriptor. Across 129 real-world modules and three backbones, TeSearch delivers the highest-quality suites of any method compared, LLM-driven evolutionary search included, and the smallest of any LLM-based one. Used as the improvement signal of a self-evolving agent, it rejects 59.9% of behavior-changing rewrites against 25.3% for the repository's tests, at a 4.0% false-alarm rate and 3.5 s per decision. The code and data are available at https://github.com/ccsnow127/TeSearch.