Web agents tested for handling changing user goals during tasks
DynaRev: Benchmarking Web Agents under Dynamic Goal Revision
Artificial Intelligence
Summary
Web agents usually get a fixed goal before they start working, but real people often change their minds while they interact with websites. The authors created DynaRev, a set of 181 tasks that add goal changes during the task at specific points, to test how well web agents can adjust. They studied different types of goal changes and measured how correct, quick, and efficient the agents were. Their tests showed agents often only complete part of the changed goals and respond differently depending on the type of revision or the model used. This benchmark helps understand and improve web agents that need to adapt when users change their minds.
What this means in practice
- •For web automation developers: Evaluate and improve web automation tools to handle users changing their goals during sessions dynamically.
- •For customer support platform teams: Test and enhance virtual assistants to better respond when customers revise their requests mid-interaction.
Authors
Shiqi Zhang, Guochen Zhou, Zesheng Wei, Hai Zhao, Zhisong Zhang
Abstract
Web agents are typically evaluated with goals fully specified before execution and fixed thereafter, whereas real users often revise their requirements during interaction. To study such interactions, we introduce DynaRev, a benchmark of 181 tasks to evaluate web agents under dynamic goal revision. DynaRev uses semantic milestones to deliver revisions at task-defined stages, covers additive revisions, unconditional overrides, and runtime-conditioned overrides, and evaluates correctness, responsiveness, and efficiency through trajectory-grounded evaluation. Five-model evaluations show that agents often satisfy only part of the task requirements, with performance varying across task sources, revision types, and models. Further analyses reveal varied milestone timing, mixed effects of dynamic delivery, and contrasting revision-handling outcomes, with gains from instruction rewriting while preserving browser state and losses from browser resets. DynaRev provides a testbed for studying how web agents adapt to evolving user goals. The code and data are available at https://github.com/zsq259/DynaRev.