Agentic judges assess long computer tasks with varied success
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
Artificial IntelligenceMachine Learning
Summary
It's hard to tell if a computer program completed a long task correctly just by looking at all the steps it took. The authors created AgentHorizon, a big set of computer tasks and their screen recordings to test how well automated 'judges' can say if a task was done right. They found that even the best AI judges make mistakes, especially on long tasks involving many steps and programs. This shows that judging complex computer work by AI still needs improvement to catch subtle errors.
What this means in practice
- •For software developers: Improve automated testing tools that verify long sequences of user actions on multiple applications without human checks.
- •For ux designers: Assess user interaction assistants more reliably to ensure they complete complex tasks correctly across operating systems.
Authors
Xing Han Lù, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, Alexandre Drouin, Siva Reddy
Abstract
Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement. Despite their flexibility, their reliability on long tasks spanning multiple applications remains unclear. A trajectory, composed of long sequences of screenshots and actions, may appear complete, but in reality violates constraints from the instruction or introduces an unwanted side effect. To identify these errors, a judge needs to examine the trajectory with respect to the user's instruction. To this end, we introduce AgentHorizon, a benchmark of 1,373 computer-use tasks (instruction-trajectory pairs) drawn from 166 hours of human-recorded trajectories spanning three operating systems. By recording trajectories for closely related instructions, we can construct negative tasks by swapping the instructions. This paired design evaluates judges on their ability to distinguish a successful trajectory from one that completed a similar (but incompatible) request. We release the benchmark under three splits: a frontier split, AgentHorizon (AH), a simplified split, AgentHorizon-Simple (AH-S), and a development split, AgentHorizon-Development (AH-D). We further evaluate eleven judges by (1) directly passing the full trajectory (with up to 300 screenshots and actions), and (2) using them as coding agents across five agent harnesses. We find that our best agentic judge, GPT-5.5, achieves 80.9% balanced accuracy on the AH subset. We find that tool-use improves certain models but results in worse performance for open-weight models, and that judges differ drastically in their ability to accept a valid trajectory and reject failed ones. Our findings highlight the need for judges that are capable of locating and verifying often hidden evidence that a task was properly completed inside long interaction histories.