Agent system better coordinates complex tasks with language models
Design and Cross-Domain Evaluation of an Agent Execution Harness
Artificial Intelligence
Summary
Completing complicated tasks with AI language models can be tricky because it needs careful planning, doing the work, and reviewing results. The authors created a system that splits the original goal from the plan and what’s been done, letting different roles plan tasks, do work, and check it before finishing. They tested this system in areas like coding, health advice, and science questions and found it improved success rates but sometimes cost more time or money. Their approach shows how organizing AI tasks carefully can help, but also when it might slow things down.
What this means in practice
- •For software developers: Use the harness to better plan, execute, and check complex coding and application-building tasks with AI language models.
- •For clinical data teams: Apply the harness to generate and review clinical responses with improved alignment to medical tasks using language-model agents.
Authors
Saurav Joshi, Manas Gupta, Ashwin Venkatesh Raman, Akshay Budhkar
Abstract
Completing complex tasks with language-model agents requires coordinating planning, execution, and review so the final result addresses the user's original objective. We present an agent execution harness that keeps the original objective separate from a revisable plan and an execution record. A planner assigns scoped tasks, workers produce outputs and artifacts, and a reviewer examines results against the original request before submitting or requesting additional work. Reusable domain-specific instructions guide these roles and are refined during construction using task inputs and execution trajectories, without official grading feedback. We evaluate complete harness configurations across terminal tasks, clinical responses, application building, and scientific questions. On Terminal-Bench 2.1, the harness passes 83 of 89 instances (93.26%); on HealthBench Professional, a Fable-configured system achieves a length-adjusted rubric score of 69.63 across 525 tasks. In a 53-task paired HealthBench comparison, the Fable-configured harness achieves an aggregate score 19.89 points higher than direct completion, despite a $-1.46$-point median task-level change; mean inference cost rises from \$0.29 to \$2.13 per task. We combine these outcomes with latency, documented instruction revisions, and reviewer decisions. Evaluating outcomes alongside execution trajectories and resource use clarifies how coordination supports task completion, where it introduces regressions, and at what cost.