Task state governance improves gui agent control and recovery
Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution
Human-Computer InteractionArtificial Intelligence
Summary
When software programs automate tasks using graphical interfaces, things can go wrong, such as unexpected changes or errors. The authors present a new way to manage these automated tasks by creating a contract that links what the task should achieve, what must stay the same, and what the software observes happening. Their system, TaskVM, shows the live task state in a way users can edit, runs those edits as real actions, checks the results from outside, and flags each step as easy or hard to undo. This approach helps users trust and control the automation better without changing the final outcome.
What this means in practice
- •For desktop software developers: Enable applications to run automated GUI workflows that users can monitor, edit, and recover more reliably.$Commercial implications: Supports building commercial software with trustworthy and recoverable automation features for end users.
- •For customer support teams: Help agents track and undo automated workflows on user interfaces to fix errors and maintain service reliability.
Authors
Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, Feng Tian
Abstract
Graphical user interface (GUI) agents now execute workflows across real applications, but when execution drifts, users must determine what changed, what should have remained untouched, and what can still be recovered. Plan-centric interfaces leave users supervising procedure, but users live with changes to application state. We introduce task-state governance, an interaction paradigm that represents a task as an editable contract linking desired outcomes, facts that must be preserved, and observed application state. We instantiate this paradigm in TaskVM, which projects live state into editable task controls, executes user edits as genuine interface actions, verifies outcomes through independent observation, and marks each action as reversible, compensatable, or irreversible. Across an automated contract benchmark, a controlled user study, and an open live-model exploration, the paradigm strengthened contract compliance and users' recovery control and trust, while execution outcomes were held constant. Task contracts open a design space for governable agent interfaces.