OptionWorld benchmark finds where AI assistants fail long tasks
OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants
Computation and Language
Summary
AI helpers often struggle to complete tasks that take days because they miss key information or fail to act on it. The authors created OptionWorld, a test that shows exactly where these assistants fall short, such as not gathering needed facts or wrongly blocking some actions. By fixing missing information or correcting misunderstandings, the assistants do better but still sometimes miss deadlines. This work helps pinpoint what AI assistants need to improve to better handle long-term tasks.
What this means in practice
- •For personal assistant developers: Diagnose and address specific failures of AI assistants managing unfinished tasks over days using OptionWorld's detailed task simulations.
- •For workflow automation teams: Improve automation tools by identifying when and why AI agents fail to complete multi-day workflows in controlled simulated scenarios.
Authors
Ruoyu Zhang, Zhaocen Liu, Weiyi Tian, Melissa Feng, Shilong Liu
Abstract
Long-term assistants must follow unfinished tasks across days as information and circumstances change. Finding information or starting an action, however, does not ensure that the task gets completed. We introduce OptionWorld, a benchmark that traces information acquisition, judgments, actions, and outcomes in simulated daily-life tasks to identify where current LLMs fall short as long-term personal assistants. In runs with archived decision inputs, the key facts remain in the model's last input before the deadline in 35.5% of failed instances. Failure analysis identifies the three most frequent problems: needed information is not obtained or combined, available opportunities to act go unused, and agents impose restrictions that the task does not require. Three follow-up experiments show that supplying missing facts restores task success in some cases; action reminders can restart work while subsequent steps still miss the deadline; and correcting a supplied note that wrongly forbids contact restores permitted inquiry. These findings locate shortcomings in maintaining task state: connecting facts to unfinished work, following through before deadlines, and separating actual restrictions from the agent's assumptions.