Language models improve computer control tasks with verbal trial learning
VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets
Artificial Intelligence
Summary
Learning how to control a computer by trying things out and learning from mistakes is hard because agents only get limited attempts. The authors study methods that use language to remember and reflect on failed tries to do better next time without changing their main programming. They test different approaches on tasks like web-based puzzles and online shopping simulations and find some methods help sometimes but hurt other times. To fix this, they create VEX², a way for the model to balance trying new things and using what it has learned, which consistently improves success across all tests.
What this means in practice
- •For ai developers: Improve language agent performance on complex computer control tasks by integrating verbal memory and exploration-exploitation balancing.
- •For automated customer support teams: Enhance virtual assistants’ ability to complete web-based tasks through trial-learning strategies that adapt within limited interaction budgets.
Authors
Yu Bai, Yukai Miao, Dawei Wang, Li Chen, Yanyu Ren, Yuqian Shi, Dan Li, Ying Xiong, Chengqiu Tan, Run Zhou, Li Li
Abstract
Learning from trial and error is a promising way to improve language agents on complex tasks such as computer control. Reflexion introduced verbal reinforcement learning, which turns failed trials into text that guides later attempts without updating model parameters. We introduce VRL-Bench, a harness for fair evaluation of trial-and-error learning under finite trial budgets. Across three models on MiniWoB and WebShop, we evaluate updates from several prominent verbal-memory methods spanning Reflexion and later work: each improves observed success over memory-free retry in some settings but reduces it in others. Replay experiments show that using reflection can reduce success rates, revealing a trade-off between exploiting experience and continued exploration. We propose VEX$^2$, a verbal exploration--exploitation scheduler that uses a language model to jointly select policies and allocate the remaining trial budget. VEX$^2$ is the only evaluated update to achieve positive observed success-rate gains over retry in all six settings.