Structured Distillation of Web Agent Capabilities Enables Generalization
2026-04-09 • Machine Learning
Machine Learning
AI summaryⓘ
The authors created a new method called Agent-as-Annotators that helps train web navigation agents using different parts of a large language model (LLM) to simulate how humans annotate tasks. They used a strong LLM (Gemini 3 Pro) to generate example tasks and fine-tuned a smaller model on the best ones. This smaller model performed better than some well-known commercial models in web navigation tasks and also worked well on new, unseen platforms. Their approach shows that carefully organizing example generation from one advanced model can produce effective, locally runnable web agents.
Large Language Models (LLMs)Web navigation agentsSynthetic trajectory generationSupervised learningFine-tuningGemini 3 ProWebArena benchmarkAgent-as-Annotators frameworkOpen-weight modelsEvaluation and ablation studies
Authors
Xing Han Lù, Siva Reddy
Abstract
Frontier LLMs can navigate complex websites, but their cost and reliance on third-party APIs make local deployment impractical. We introduce Agent-as-Annotators, a framework that structures synthetic trajectory generation for web agents by analogy to human annotation roles, replacing the Task Designer, Annotator, and Supervisor with modular LLM components. Using Gemini 3 Pro as teacher, we generate 3,000 trajectories across six web environments and fine-tune a 9B-parameter student with pure supervised learning on the 2,322 that pass quality filtering. The resulting model achieves 41.5% on WebArena, surpassing closed-source models such as Claude 3.5 Sonnet (36.0%) and GPT-4o (31.5%) under the same evaluation protocol, and nearly doubling the previous best open-weight result (Go-Browse, 21.7%). Capabilities transfer to unseen environments, with an 18.2 percentage point gain on WorkArena L1 (an enterprise platform never seen during training) and consistent improvements across three additional benchmarks. Ablations confirm that each pipeline component contributes meaningfully, with Judge filtering, evaluation hints, and reasoning traces each accounting for measurable gains. These results demonstrate that structured trajectory synthesis from a single frontier teacher is sufficient to produce competitive, locally deployable web agents. Project page: https://agent-as-annotators.github.io