Skill focused web agents improve task success on unseen websites
Back to Basics: A Skill-Centric Approach to Generalist Web Agents
Computer Vision and Pattern Recognition
Summary
Web browser agents that act like people often fail because they don't practice the right skills. The authors built MiniWeb, a small but diverse web environment with real tasks and detailed labels for basic web actions. They used this to find which skills agents struggle with, especially drag and gesture actions. By training agents directly on these skills and then combining them to complete tasks, they improved how well agents perform on new, unseen websites and other benchmarks. This work shows that practicing key web skills separately helps agents handle more complex web tasks.
What this means in practice
- •For software engineers: Build smarter browser automation tools that learn and combine core web interaction skills to handle new sites reliably.
- •For ux designers: Assess and improve web interface elements by identifying which interaction skills are hardest for automated agents to perform.
Authors
Minh Pham-Dinh, Md Farhan Ishmam, Hernan Yupanqui, Reaz Hassan Joarder, Kenneth Marino
Abstract
Browser agents do not just need more practice. They need to practice the right things. We introduce MiniWeb, a realistic miniature of the web comprising 65 interconnected websites across 15 domains, where agents navigate, search, and transact as people do. Its 369 human-annotated tasks cover everyday workflows and label 37 reusable interaction primitives grouped into 7 skill families, allowing us to diagnose which skills fail rather than only whether a task succeeds. Benchmarking ten open-weight agents, we find that drag & gesture is consistently the weakest family, with no agent solving more than 24% of its instances. To address this gap, we introduce MiniWebGen, a framework that autonomously generates verified, skill-centric training data from MiniWeb. Fine-tuning Qwen3.5-4B on just 3.1K verified trajectories improves single-primitive success by 24.9pp across 13 unseen websites, with gains across every skill family. However, these trajectories teach short-horizon interactions, which do not directly translate to full-task success. We build on this result with MiniWebAgent, which pairs the trained executor with a planner that decomposes full tasks into the skill-level steps it learned to perform. This approach improves task success by 19.4pp points on unseen MiniWeb sites and transfers to external benchmarks, with gains of 9.3pp on WebArena-Lite, 16.1pp on Online-Mind2Web, and 20.0pp on WebVoyager, without training on their trajectories. We provide a simple recipe for compact browser agents: practice reusable skills directly, then compose them to solve unfamiliar tasks.