Plug-in decision heads improve web agent actions with fewer tokens
Why Generate a Click? Plug-in Decision Heads for Web Agents
Computation and Language
Summary
Web agents that use large vision-language models usually decide actions by generating text commands like clicking buttons. This paper shows a way to have smaller parts, called decision heads, read the model’s internal understanding directly to decide actions more efficiently. The authors found that these decision heads can sometimes match or improve on the model’s performance while generating fewer words. This approach also avoids hurting the model’s ability when trained separately and works well across several test web navigation tasks.
What this means in practice
- •For web automation developers: Use decision heads to speed up web navigation agents by reducing text generation while maintaining accuracy on complex web tasks.
- •For software engineering toolmakers: Incorporate frozen model decision heads to improve the efficiency and reliability of automated user interface testing tools.
Authors
Chengguang Gan, Yunhao Liang, Hanjun Wei, Qinghao Zhang, Zhixi Cai, Shiwen Ni
Abstract
Web agents built on vision language models decide every action by generating text such as click(12), although most of that text encodes a choice among a few action types and a few dozen marked elements. We read these choices directly from the hidden states that the generator computes while it reads the goal and the screenshot. Plug-in decision heads on a frozen generator predict the action type, pool the visual tokens inside each numbered box to pick the target, and point at a listed text option or at a WRITE slot that leaves the writing of new text to the generator. A confidence cascade executes the heads' action when they are sure and otherwise lets the generator decode from the KV cache of the same forward pass. With Qwen3-VL 4B and 8B generators trained on 213k web steps, a strict threshold keeps the cascade between 0.3 points below and 1.4 points above the generator on seven offline test sets while the heads decide 12 to 64% of the steps. A looser threshold raises step success on unseen Mind2Web websites by 3.1 and 1.8 points, costs up to 2 points on GUIAct and MiniWoB++, and lifts WebArena task success from 12.5 to 13.8% (4B) and from 13.6 to 14.6% (8B) with less than half of the generated tokens. Training the heads jointly with the generator lowers the generator's own step success by up to 4.9 points; heads on a frozen generator avoid this cost.