Agent performance depends on both language model and harness used
Finding the Right Fit: Model-Harness Interactions across Agent Tasks
Artificial IntelligenceSoftware Engineering
Summary
Choosing how to run an AI agent means selecting both the language model and the system that controls it, called a harness. The authors found that the best model doesn't always do best with every harness, and the best pair changes depending on the tasks. They showed that some models work better with simpler harnesses, while others need more error-tolerant systems. Their work suggests that agents should be tested as a combination of model, harness, and task to find the best fit.
What this means in practice
- •For application developers: Select optimal model and harness pairings based on task to improve AI agent performance and reduce operational costs.
- •For cloud service providers: Optimize AI service configurations by matching language models with compatible harnesses to deliver cost-effective and reliable agent tasks.
Authors
Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng, Zhiyi Lyu, Xuyu Dong, Feng Chen, Bo An
Abstract
Choosing an agent system means choosing both a language model and the harness through which it acts. We ask whether a strong model, harness, or pairing stays strong when the setting changes. We evaluate 66 configurations: four configurable harnesses (OpenHands, DeepSeek Harness, PI, and openJiuwen) paired with five models on TUA-Bench, ALE-CLI, and Terminal-Bench 4, plus the native Codex-GPT and Claude Code-Claude pairings. Model rankings reverse across harnesses. On Terminal-Bench 4, Claude leads GPT by 7.94 points in OpenHands but trails it by 30.16 points in PI. For four of the five models, the best harness changes from one benchmark to another, yet some pairings hold: openJiuwen gives Kimi its highest score on all three benchmarks, by 5.61 to 11.11 points. A model's own vendor harness is not reliably its best, and higher cost does not reliably buy a higher score. On Terminal-Bench 4, GPT scores higher under PI than under DSH at less than a quarter of the cost per task. Matched trajectories suggest why fit varies. Models start almost all repairs themselves, so much depends on whether the harness hands failures back in a form the model can use. GPT does best with PI's lean scaffold, while Kimi, which often issues malformed tool calls, does best in openJiuwen. We argue that the model, the harness, and the task should be evaluated together, and we release the harness adapters, evaluation code, and all 6,204 scored trajectories at https://github.com/liyix/finding-the-right-fit and https://huggingface.co/datasets/yixuanli97/finding-the-right-fit.