Specforge improves coding agents by refining code behavior descriptions
SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications
Artificial IntelligenceSoftware Engineering
Summary
Coding agents that write programs can do better by improving their instructions without changing their underlying knowledge. The authors look at how these agents can repeatedly fix and enhance the descriptions of what the code should do, based on testing results. They deal with the challenge that changing one part can cause problems elsewhere by using a new search method that carefully chooses what to improve and how. Their approach called SpecForge helps these agents reliably make better code and also helps them with other programming tasks.
What this means in practice
- •For software development teams: Improve automated code generation by refining documentation-driven behavior specifications to produce higher-quality, tested programs.
- •For automated testing teams: Use enhanced behavioral specifications to better detect and correct code errors during test-driven development processes.
Authors
Yutong Cheng, Haifeng Chen, Peng Gao, Wei Cheng
Abstract
Coding agents based on frozen language models can improve their performance by refining the external context they rely on, without updating model weights. We study recursive self-improvement through behavioral specifications: natural-language code documentation that describes the behavior of program entities and guides agents in generating correct code. We formulate a self-improvement loop in which an agent generates code from its current specifications, evaluates it with execution-verified tests, and refines the specifications based on observed failures. The improved specifications then guide subsequent generation, progressively improving the agent's understanding of the codebase. However, recursive refinement is challenging because program entities are interdependent. Improving the specification of one entity can inadvertently degrade the correctness of others, a phenomenon we call output coupling. This creates a whack-a-mole effect in which local improvements cause regressions elsewhere, preventing consistent progress. We introduce SpecForge, a multi-frontier tree search framework that addresses this challenge with three mechanisms: (i) Pareto-guided state selection preserves complementary specification states and prevents premature commitment; (ii) dependency-constrained entity selection prioritizes callee-before-caller refinement; and (iii) error-diversified expansion explores multiple refinements using different subsets of test failures. On DevEval+, SpecForge achieves the best performance across five backbone models, including a 94.9% solve rate with Claude-4.5-Sonnet and a 5.7-percentage-point average improvement over the strongest baseline. The improved specifications also make the agent better at tasks outside the loop: they transfer to cross-language translation and to new-feature implementation, where they outperform the benchmark-provided requirement documents.