Computer Use Agents improve learning with real-time feedback
ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Using computers often involves many steps, but teaching computer-use agents how to perform tasks can be hard because they only get feedback at the end. The authors created a method called ComputerSD that gives these agents feedback at every step based on what they do in real time. This helps them learn better decisions during tasks, not just at the end. Tests show that ComputerSD works better than older methods, even when used in new or unexpected situations.
What this means in practice
- •For software developers: Improve the online training of interactive agents that use graphical user interfaces by integrating real-time feedback for better performance.
- •For customer service automation teams: Enhance chatbot or virtual assistant responsiveness by adopting self-distillation methods that learn from step-by-step GUI actions.
Authors
Yong Du, Tongbo Chen, Zhengxi Lu, Yizhou Liu, Bofan Chen, Tao Jiang, Wenhao Xu, Yongliang Shen
Abstract
Online training enables computer-use agents (CUAs) to improve through interaction with executable environments. However, existing methods primarily rely on sparse outcome rewards, which provide no supervision for intermediate actions. On-policy self-distillation (OPSD) offers token-level learning signals through privileged rescoring, but directly applying it to CUA online training presents two challenges: fixed guidance may become misaligned with the student's current state, and guidance-induced probability shifts may conflict with step-level correctness. We introduce ComputerSD, an online self-distillation method for CUAs that converts real-time feedback from executed GUI transitions into guidance for policy learning. A fine-tuned GUI analyzer produces guidance and a step-level value score after each action; the guidance provides privileged context, while the score regulates the resulting OPSD signals. ComputerSD jointly optimizes token-level OPSD and trajectory-level GRPO in a fully asynchronous training framework. On OSWorld-Verified, ComputerSD outperforms outcome-only GRPO by 1.9 and 4.1 percentage points on the general-purpose Qwen3-VL-8B-Thinking and specialized EvoCUA-8B backbones, respectively. Evaluation in out-of-distribution settings further supports the generalizability of ComputerSD. These results demonstrate the effectiveness of learning from real-time feedback through online self-distillation for CUAs.