Harness primitives improve language model task handling at test time
Composing Task-specific Agent Harnesses at Test Time with Reusable Primitives
Artificial Intelligence
Summary
Large language models use harnesses to manage how they gather information, use tools, and finish tasks. The authors found that a fixed harness doesn’t work well for every task because different tasks need different harness mechanisms. They created reusable harness pieces called Harness Primitives and a system named STITCH that picks and combines these pieces for each task without rewriting code. This approach improves success rates on tasks and is much faster than creating new harnesses from scratch.
What this means in practice
- •For ai product developers: Create AI agents that adapt their internal control mechanisms to diverse tasks for better performance without significant overhead at run time.
- •For automation engineers: Build more reliable automated workflows by combining tested harness components tailored to specific operational tasks.
Authors
Peng Kuang, Haibo Jin, Dehao Wu, Feiyang Deng, Xiaopeng Yuan, Jerry Wang, Haohan Wang
Abstract
Agent harnesses govern how large language models (LLMs) gather context, invoke tools, verify results, preserve state, and terminate, largely affecting agent performance. However, the value of each harness mechanism can differ across heterogeneous tasks: a mechanism that improves one task may impose overhead or context distraction on another, leading to the suboptimality of a global harness. We characterize this suboptimality as a mismatch induced by fixed mechanism choices, motivating task-specific harness construction. Nonetheless, generating harness code for each task introduces generation and debugging costs, with execution risks that can compound as more mechanisms are generated. To address those challenges, we introduce Harness Primitives, reusable harness mechanisms with clear application scope and composition contract mined from failed task trajectories. Based on Harness Primitives, we propose STITCH, a framework that Selects suitable primitives given Task Information and compiles them into Task-speCific Harnesses at test time. This separation enables task-specific harnesses without generating or repairing mechanism code at test time. Extensive experiments demonstrate that STITCH not only improves harness adaptability and robustness, but also scales with the primitive library size, boosting task success rates by up to 12 points over fixed harness baselines, surpassing human-designed harnesses like Codex CLI while maintaining a minimal test-time harness composition overhead of only 2.7%, 638 times more efficient than generating task-specific harnesses from scratch. Ultimately, our work demonstrates that building task-adaptive harnesses can be beneficial for completing diverse tasks and that building reusable primitives can be a promising path towards this goal.