Memory systems struggle to improve coding agents on real software tasks
VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
Software EngineeringComputation and Language
Summary
Coding agents that help fix or improve software can learn from past experiences stored in memory systems. The authors created a new benchmark, VibeMemBench, to test if these memory systems actually help coding agents do better on real coding tasks from existing software projects. They found that when past experiences are directly inserted, some agents improve a bit, using fewer steps to solve problems. But when current memory systems try to build and use these experiences on their own, they mostly fail to outperform having no memory system at all. This reveals a big gap between the useful past experience available and how well memory systems use it.
What this means in practice
- •For software engineering teams: Evaluate and improve memory systems in development tools to better reuse past coding experiences from repositories for fixing bugs and adding features.
- •For software quality assurance teams: Use VibeMemBench to measure how memory support affects the efficiency and success of automated coding agents in software maintenance tasks.
Authors
Liyang Fan, Yingcheng Shi, Yongbin Li, Chenghao Sun, Xin Chen, Xander Xu, Hu Wei, Shiwen Ni, Min Yang, Jieping Ye
Abstract
Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluations do not show whether those systems improve executable repository work. Repository benchmarks test code changes but do not isolate memory, while memory benchmarks score recall without measuring downstream coding outcomes. We introduce VibeMemBench, a benchmark for evaluating memory systems on 111 coding targets from 90 SWE-rebench V2 repositories and 3,634 history trajectories from the target repositories. The targets follow the SWE benchmark style and cover bug fixes, feature requests, interface changes, and configuration work. An agent edits each target codebase under a declared memory condition. Executable tests decide task resolution. Each target is retained only when injected history experience improves its executable outcome in a reference setting, so every target carries a prior experience whose usefulness is verified by execution in that setting. The frozen verified experience is then transferred to five held-out solvers. Direct injection raises observed task resolution on four of them by 1.1 to 4.5 percentage points while lowering agent steps on all five. Yet when four existing memory systems must construct and retrieve experience from the same history, eleven of twelve solver and system pairings fail to exceed the matched memory-off baseline. VibeMemBench exposes the gap between the useful experience that repository history holds and the experience existing memory systems deliver for repository coding tasks.