SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories
2026-08-10 • Software Engineering
Software Engineering
AI summaryⓘ
The authors study how coding agents respond to software requirements that evolve over time. They find that these agents may produce different programs even when the final requirements are the same, depending on the order in which the requirements were given. They call this 'specification-path sensitivity' and propose a test called SpecPath to measure it. Their experiments show that success on a final set of requirements doesn't guarantee consistent behavior across different revision histories. This means it's important to check if agents truly understand evolving specifications, not just the final version.
coding agentssoftware requirementsspecification-path sensitivityspecification revisionactive-contract resolutionSpecPathsoftware testingprogram synthesisrequirement historiesagent robustness
Authors
Yangfan Wu, Haozhe Wang, Huanyu Yang, Jianmin Ji, Fangzhen Lin
Abstract
Modern coding agents increasingly appear capable of following complex software requirements, yet their success leaves a critical ambiguity: do they resolve the active specification, or merely follow the most salient path by which it was stated? We identify specification-path sensitivity, a failure mode in which requirement histories that are equivalent in their final meaning lead the same agent system to produce behaviorally different programs. This reframes evolving-requirement evaluation as active-contract resolution: before writing code, an agent must determine which requirements still count. Building on this view, we introduce SpecPath, a diagnostic evaluation that holds the repository, final contract, verifier, agent system, and execution budget fixed while changing only the revision path that leads to the contract. Rather than treating each patch as an isolated pass or failure, SpecPath uses paired executable outcomes to reveal whether an agent realizes the same tested behavior across contract-equivalent histories. Across five calibrated software tasks and fourteen coding-agent configurations, aggregate direct and revision-history accuracy is nearly unchanged; nevertheless, 35 of 100 complete blocks that succeed on the direct specification fail on at least one equivalent history. These results show that implementation success on a consolidated request does not guarantee specification-path invariance. Evaluating evolving requirements therefore calls for controlled tests of whether agents are robust to the path by which a specification becomes final.