Deskforge improves computer use agents with dense desktop annotations
DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Computer programs that help users by interacting with desktop computers need to understand exactly what they see on complex screens. The authors created DeskForge, a tool that mixes real applications on a desktop and creates detailed labeled images showing what every button and window is. They collected a big dataset from DeskForge and used it to train models that got better at recognizing parts of the screen and completing tasks involving multiple steps. Their results show this method helps computers understand screens better and do more complex work reliably.
What this means in practice
- •For software developers: Train AI assistants to accurately identify controls and windows in complex desktop setups for better automation and support.
- •For automated testing teams: Use dense annotated desktop environments to improve the reliability of automated UI testing bots on varied software interfaces.
Authors
A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys, Peter W. J. Staar
Abstract
Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. The framework code, the dataset, and the fine-tuned model are available from the project page: https://saidgurbuz.github.io/deskforge/