Offline benchmark reveals guide sentences improve web agent actions
Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents
Computation and Language
Summary
Web agents try to interact with websites by following instructions and taking actions, but testing them live is unreliable. The authors created a new offline test setup that uses past successful interaction data to evaluate agents consistently. They found that providing a related guide sentence alongside the action helps the agent choose elements better, and this effect grows with bigger models. The guide works like a meaningful instruction, not just a label, improving action accuracy significantly.
What this means in practice
- •For web automation developers: Build web agents that use paired guide sentences to improve element selection accuracy without needing live environment tests.
- •For language model engineers: Fine tune language models offline to jointly generate guides and actions, enhancing their performance on web interaction tasks.
Authors
Chengguang Gan, Yunhao Liang, QingHao Zhang, Shiwen Ni
Abstract
Web agents are usually evaluated in live environments, where environment state and judge models drift between runs, so the same checkpoint rarely reproduces the same score, making controlled studies of training phenomena impractical. We present WebMRE, an offline benchmark of 541 tasks and 5,293 steps derived from successful WebArena trajectories, with fully audited test labels and a deterministic protocol that scores a checkpoint identically on every run without any environment. Each step pairs a human oriented guide sentence with a grounded action, enabling the first study of the mutual reinforcement effect between them in web agents. Averaged over three seeds the effect holds for both models in both decoding orders and grows with scale: jointly decoding a guide lifts element selection over an action only reference by 0.9 and 0.2 points for Qwen3.5-4B and by 1.7 and 2.2 points for Qwen3.5-9B. A mediation analysis shows that the guide is a causal channel rather than commentary: forcing the gold guide as a decoding prefix lifts action accuracy from .422 to .684, another step's guide collapses it to .055, and a paraphrase that renames the target still recovers half of the gain, so the channel carries instruction meaning and not only the label string. The same channel yields an offline reward that only a replayable protocol makes computable, though optimizing it from a strong checkpoint brings no gain yet. Our fine tuned models outperform GPT-5.5, Claude Opus 4.8, and Gemini 3.5 Flash, run zero shot, on every offline metric.