Process based evaluation reveals how computer use agents fail tasks
OSWorld-Pro: Process-based Evaluation for Computer Use Agents
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Sometimes intelligent computer helpers make mistakes, but just checking their final answers doesn’t show where or why they went wrong. This paper introduces OSWorld-Pro, a big set of tasks broken into many small steps with detailed human notes that help judge each step. Using special AI judges aligned with humans, the authors can see exactly when and how these computer helpers get stuck or make errors while working. They found current AI models still struggle with many of these smaller steps, which helps guide future improvements.
What this means in practice
- •For software developers: Develop and test smarter AI assistants that interact with computer interfaces by diagnosing exact points of failure during task steps.
- •For automation engineers: Improve reliability of automated computer-use systems by using fine-grained error data from OSWorld-Pro’s stepwise evaluations.
Authors
Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, Yi Dong
Abstract
Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.