HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
2026-08-03 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors study how the design of a "harness"—a setup that helps large language models (LLMs) understand and act in tasks—impacts their performance. They point out issues in previous automatic harness improvement methods, like overfitting and mixed-up component updates. Their new method, HarnessCompass, fixes these by applying global rules, getting richer feedback from the model itself, and optimizing each part separately before combining them. This approach makes the harness better and more general, improving results on tests and working well with different tasks and models.
large language modelsharness designautomatic harness evolutiontrajectory-derived signalscomponent-wise optimizationoverfittingSWE-benchPass@1GPT-5.4agent-environment interaction
Authors
Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen, Yuhang Tian, Jun Yang, Huipeng Ma, Chenhao Li, Guangyuan Feng, Xudong Li, Yizhou Jin, Yan Xu
Abstract
Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54\% to 66\% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.