HarnessPro improves AI test performance while lowering costs
HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs
Machine LearningComputation and Language
Summary
Building good tests for AI programs helps them get better at tasks. The paper presents HarnessPro, a way to improve these tests so AI solves more problems but uses less computing power. The authors do this by making different improved tests that focus on speed, quality, or a balance of both. They use large language models to create and check tests without needing official answers. Their method works well across several programming benchmarks and different AI models.
What this means in practice
- •For ai development teams: Generate and refine tests automatically to improve AI code agents’ success rates while controlling compute cost for different deployment needs.
- •For software engineering tool builders: Create tools that evolve testing harnesses balancing thoroughness and efficiency for automated code evaluation without relying on ground-truth answers.
Authors
Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen, Honglin Lin, Junru Lu, Zhongxia Chen, Shuo Yang, H. Vicky Zhao, Jie Tang
Abstract
Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro