EvoOtter: Evolutionary Reproduction Test Generator
2026-07-03 • Software Engineering
Software EngineeringMachine Learning
AI summaryⓘ
The authors address the challenge of creating tests that reproduce software bugs, which is hard because bug descriptions are often unclear. They propose a new method called EvoOtter that uses evolutionary programming combined with large language models (LLMs) to generate these tests more efficiently and accurately. EvoOtter reduces costs by smartly managing test runs and LLM calls with techniques like successive halving and batched crossover. Their approach produces better-quality bug reproduction tests much cheaper than previous methods. This work shows how evolutionary algorithms and LLMs can work together effectively in software engineering.
bug reproduction testlarge language modelsevolutionary programmingsuccessive halvingbatched crossovertest execution feedbacksoftware testingcode mutationfitness scoreinference scaling
Authors
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, Martin Hirzel
Abstract
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using large language models to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with large language models for software engineering.