Agent memory improves by matching actions with page responses
Action Conditioned Bisimulation For GUI Agent Memory
Artificial Intelligence
Summary
It can be tricky for a software agent to remember web pages correctly because pages that look alike might respond differently to clicks. The authors came up with a new way for the agent to decide when two pages are really 'the same' by checking if they behave the same way after actions, not just if they look similar. Using this approach, the agent gets better at tasks on web interfaces without needing to train on appearance. This method was tested and showed improvements in a benchmark called MiniWoB++.
What this means in practice
- •For web interface developers: Improve automated agents that learn user interactions by reliably recognizing functionally same pages regardless of visual similarity.
- •For robotics software teams: Enhance robot memory systems for environments where visual states may be ambiguous but action outcomes differ.
Authors
Hongbo Zhang, Liuyang Song, Quanquan Li, Daqian Yang, Yan Wen, Zhengtao Yao
Abstract
An agent that remembers what it did on a web page must decide when two pages count as the same. Memories built on observation similarity merge pages that look alike but behave differently, and GUIs are full of such pages: two tabs of one widget or two rows of one menu answer the same click differently. We define the merge rule as an action-conditioned bisimulation over the empirical predictive state graph a frozen agent fills as it acts. Two states merge only when their shared actions lead to agreeing outcomes and successor blocks under an affordance label. Observation similarity never enters the rule, and nothing is trained. It replaces the merge rule of an existing outcome-value memory, so a closed-loop comparison isolates it. On MiniWoB++ it raises success rate over a memoryless agent, while a control taking identical exploratory detours, the prior successor-representation merge, and the same criterion without action conditioning change nothing.