Adaptive Strategy Generation for Boundary Value Exploration Beyond Numeric Inputs
Software Engineering
Summary
The authors developed ABEX, a system that uses language-based AI agents to explore software boundaries where program behavior changes abruptly. Unlike previous methods that rely on manually designed mutation operators for specific input types, ABEX creates and adapts strategies on its own to find these boundaries across various input types like numbers, strings, and arrays. Their tests show ABEX performs better at finding important boundaries and faults in software, especially for non-numeric inputs which were not addressed before. The authors also found that the key to ABEX’s success is its ability to generate strategies dynamically based on feedback.
Authors
Sabinakhon Akbarova, Felix Dobslaw, Robert Feldt
Abstract
Software behavior often changes abruptly at boundaries between input regions, and these transitions are known to be fault-prone. Boundary Value Exploration (BVE) automates boundary discovery by searching for pairs of similar inputs that nevertheless trigger different program behaviors. Existing automated BVE techniques rely on mutation operators hand-engineered for each input type, or even for each function under test, which has confined their use to numeric inputs. We present ABEX, an agentic LLM-based framework that replaces operator engineering with adaptive strategy generation: specialized LLM agents propose, select, and execute boundary-exploration strategies, guided by execution feedback and a quality-diversity (QD) archive. Because strategies are expressed in natural language, they can encode both type-level and function-specific knowledge, and effective strategies can even be stored and reused. We evaluate ABEX in a black-box setting on 20 functions with numeric, string, array, and mixed inputs. On numeric functions, ABEX outperforms a state-of-the-art QD method on 10 of 11 functions, with average QD-scores 11.7x higher. On non-numeric functions, addressed here for the first time in automated black-box BVE, ABEX discovers domain-aligned boundary behaviors for all subjects. Mutation testing shows the discovered boundaries are fault-revealing: with equally sized test suites, ABEX reaches an average mutation score of 86.2% versus 61.9% for the QD baseline, and kills nine times as many hard-to-detect stubborn mutants. An ablation study identifies adaptive strategy generation as the primary driver of these gains.