Plover: Steering GUI Agents through Plan-Centric Interaction
Artificial Intelligence
Summary
The authors study how computer programs that control graphical user interfaces (GUIs) often fail when the layout changes or unexpected windows pop up. They created Plover, a system that shows users the plan the program follows to complete tasks on the screen, making it easier for users to watch and fix mistakes. Users can edit the plan directly, guide the program with natural language, and intervene using screenshots, all without losing previous progress. Their tests show that making the plan visible and editable helps users fix problems more easily and makes the automation more flexible and understandable.
Authors
Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, Dongyu Liu
Abstract
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.