Benchmark measures software setup skills of AI coding assistants
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Software EngineeringArtificial Intelligence
Summary
The paper looks at how well large AI language models can manage complex software setups, called harnesses, when working on long tasks. The authors created a new test called Doc2Harness to see how good these models are at building and using these harnesses correctly. They tested many advanced models and found that even the best one made mistakes in about half the cases. The hardest problems involved organizing different parts of the software correctly across modules. This benchmark is the first reliable way to measure this specific skill of AI coding agents.
What this means in practice
- •For software development teams: Evaluate and improve AI tools for managing complex software environments during coding tasks.
- •For devops engineers: Use the benchmark to test automation scripts that configure and run isolated software containers reliably.
Authors
Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Jiayu Yao, Jiafeng Guo, Xueqi Cheng
Abstract
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.