Agent harnesses pruned to cut token costs without losing power
SHarP: Saliency-based Pruning of Agent Harnesses
Artificial Intelligence
Summary
Agent harnesses are like toolkits that help large language models handle complicated jobs by combining instructions and tools. The group behind this paper found that many parts of these toolkits are unnecessary and only make the system slower and more costly to run. They created a way to measure how important each part is and safely remove the less useful ones. This pruning leads to simpler toolkits that work just as well but use fewer resources. Their findings suggest many agent harnesses can be made leaner without hurting performance.
What this means in practice
- •For ai platform developers: Reduce token use and speed up complex AI task workflows by pruning unnecessary harness modules identified through saliency.
- •For enterprise automation teams: Streamline AI-driven automation systems by removing redundant components in agent harnesses while maintaining task effectiveness.
Authors
Xinyi Gao, Qiucheng Wu, Kaizhi Qian, Handong Zhao, Shiyu Chang, Yang Zhang
Abstract
Agent harnesses are systems that coordinate model calls, tool use, and task execution to help large language models complete complex tasks. To meet task requirements and address failures, these systems are often iteratively refined by amending and patching their instructions, tools, and workflows, continuously increasing harness complexity. It is therefore unclear whether some resulting harness modules are redundant, introducing substantial token overhead with little, if any, performance gain. Inspired by neural network pruning, in this paper, we study harness pruning as a means of striking a better balance between task performance and token cost. We propose SHarP (Saliency-based Harness Pruning), a simple yet effective pruning strategy based on the saliency of each harness module with respect to performance and efficiency. Specifically, we first identify tools, instructions, and supporting mechanisms as components that can be individually disabled. We then estimate the saliency of each module by ablating it and assessing its task performance and token cost relative to the full set of single-module ablations. Modules with the smallest contribution to performance or largest computational overhead are subsequently pruned. Our evaluation across various harnesses on held-out validation sets reveals a surprising finding: most harnesses that we studied are highly redundant and can maintain comparable performance and efficiency even after a substantial portion of their modules are pruned. Our pruning approach and empirical findings provide new perspectives on agent harness design and optimization.