Vestrum improves language model agents by adapting their control software
Vestrum: Improving Agent Harnesses by Adapting Their Verification, Structure and Memory
Artificial Intelligence
Summary
Language models often rely on software called agent harnesses to decide how they gather information, use tools, remember things, and check their own work. Improving these harnesses is usually slow and expensive because it takes a long time to test them in real situations. The authors introduce Vestrum, a system that identifies where these harnesses fail and suggests targeted fixes without retraining the main language model. By learning from past mistakes and adapting the harnesses’ verification, structure, and memory, Vestrum boosts performance on various tasks without extra training cost.
What this means in practice
- •For software developers: Improve language model-driven applications by automatically fixing agent control software based on error patterns without retraining models.
- •For data annotation teams: Enhance cell-type annotation workflows using adaptive memory and verification improvements to increase agreement accuracy on biological datasets.
Authors
Jayant Parashar, Eugene F. Douglass, William C. Bastian, Suchendra M. Bhandarkar
Abstract
An agent harness controls how a language model accesses information, uses tools, preserves memory, and checks its work. Improving this software is costly when each evaluation requires a long interaction with an environment. We introduce Vestrum, a framework that turns failures in execution traces into scoped harness changes without training the task model. Its organizing overhypothesis is that tasks of a shared kind may exhibit recurring failures whose remedies transfer within that kind. Vestrum expresses failures as recognizable classes, proposes changes across verification, retrieval, decomposition, and knowledge synthesis, and screens their scope before evaluating them as a bundle. A persistent lessons file informs subsequent proposals. Across five settings and two baseline harnesses, the frozen harnesses improve held-out performance: UltraHorizon rises from 47.6 to 59.8 over GAM, Terminal-Bench 4 Hard from 63.7% to 70.3% of checks passed over Claude Code on eight held-out tasks at 1.03x test cost, and cell-type annotation agreement from 67.5% to 77.8% on held-out sections of one slide, alongside gains on LoCoMo and AMA-Bench. Across our searches, verification grounded in evidence helped both intermediate steps and final answers, at lower cost at intermediate steps, while critics asked to rebuild finished answers broke more than they repaired. On the three memory benchmarks, Vestrum also scores above the evaluated GEPA configurations in every paired evaluation.