Mid-harness improves terminal agent success with smarter action verification
Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
Computation and LanguageArtificial IntelligenceMultiagent Systems
Summary
Terminal agents try out actions in a computer system, but sometimes their actions fail and mess things up even when better options exist. The authors studied whether checking multiple possible actions before picking one to execute could help. They created Mid-Harness, a system that samples actions and verifies them before running one, using the same underlying models. Their tests showed that a strong verifier significantly improves success rates, especially when checking several actions, making the agent more reliable without changing the original action generator.
What this means in practice
- •For software engineers: Improve automated command execution tools by integrating smarter verification of generated commands before they are run.
- •For systems reliability teams: Enhance robustness of terminal automation by reducing failed or harmful actions through action candidate verification.
Authors
Minki Kang, Ryo Hachiuma, Shaokun Zhang, Subhashree Radhakrishnan, Yonggan Fu, Jindong Jiang, Mingjie Liu, Ehsan Hosseini-Asl, Yi Dong, Yu-Chiang Frank Wang, Byung-Kwan Lee
Abstract
Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.