Pairing coding agents and tools greatly affects their success and efficiency

Pairing Matters: Performance, Failures, and Efficiency in Lightweight Coding Agents

Software Engineering

Summary

It matters a lot how lightweight coding AI models are paired with the software tools (called harnesses) that run them. The authors show that one model can perform very differently depending on which harness it uses. They tested many combinations and found that failure reasons and efficiency vary widely. This means picking the right harness and model pair is very important to get the best results when deploying coding AI agents.

What this means in practice

  • For enterprise software engineers: Select the best combination of coding models and harnesses to maximize task resolution and minimize failures in proprietary code environments.
  • For ai tool integrators: Design or tune agent harnesses informed by detailed failure patterns to improve efficiency and reliability of lightweight coding agents.

Authors

Lu Zhao, Christopher M. Poskitt, David Lo

Abstract

Self-hosting coding agents is often necessary for software-engineering teams that must keep proprietary code and development traces within controlled infrastructure. Lightweight, open-weight models make such deployment more feasible by reducing hardware requirements, while strong results on repository-level benchmarks demonstrate their potential for demanding coding tasks. Deploying these models as coding agents requires pairing them with an agent harness, making their joint performance a key deployment consideration. Yet existing leaderboards primarily report fixed harness-model configurations rather than controlled comparisons across alternative pairings. To address this gap, we investigate harness-model interactions through a controlled full-factorial evaluation of six open-source harnesses and three lightweight coding models across 100 SWE-bench Verified tasks, producing 1,800 graded runs under common serving, metering, and grading conditions. For the same model, resolution rates differ by up to 44 percentage points across harnesses, while harness rankings change across models. We apply an evidence-based failure taxonomy to 1,100 unsuccessful runs and find substantial differences in failure patterns across harness-model pairings. A model's dominant failure modes under one harness need not dominate under another. Efficiency is similarly dependent on the pairing: even among model-directed loop harnesses, charged tokens per resolved task vary by more than 20-fold. These findings make harness-model selection a central deployment decision for lightweight coding agents. Strong performance with one harness or model does not guarantee similar results with another. We translate these findings into practitioner guidance for pairing selection, token-efficiency assessment, and trajectory-informed harness engineering, and release the complete experimental records and analysis artifacts.