Agent s4 evolves better interfaces for computer-use tasks
Evolving Agent Interfaces
Artificial Intelligence
Summary
Computer-use agents often struggle because humans design the interfaces they work through, which can be inefficient. The authors present Agent S4, a system that learns to create and improve its own interfaces by watching how agents perform tasks and evolving these interfaces over time. This approach helps agents use applications more effectively, improving their success rates compared to fixed interfaces. Experiments showed that Agent S4 outperformed other advanced models while using fewer resources.
What this means in practice
- •For software developers: Improve automation tools by integrating evolving interfaces that adapt to new workflows without manual redesign.
- •For enterprise ai teams: Enhance AI agents’ ability to interact with complex software by evolving their interfaces for higher task success rates.
Authors
Saaket Agashe, Yuzhe Yang, Gonzalo Gonzalez-Pumariega, Bhuvan Chandra Koduru, Chih-Lun Li, Jiachen Yang, Ang Li, Xin Eric Wang
Abstract
Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function. Both the interfaces and the knowledge of when to use each are built by hand, which cannot keep pace with new applications, workflows, and models. We argue that interface design should itself be an agent capability. We introduce Agent S4, an autonomous computer-use scientist that builds Evolving Agent Interfaces from task experience. A meta-agent examines task agents' trajectories on synthetic training tasks, without success labels, and constructs and revises interfaces and usage knowledge in a repository. Population-based evolution then selects which repositories to develop further by their performance on validation tasks the meta-agent never sees, favoring changes that generalize beyond the seen tasks. On OSWorld V2, Agent S4, with Qwen3.8-27B and the evolved repository of interfaces, sets a new state of the art in open-source computer-use agents, reaching 21.3% binary score, 3.8 times the same model with fixed interfaces. This is above both GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs. On ScienceBoard, the evolved interfaces improve the success rate from 41.4% to 58.6% with the same model. These results suggest that useful interfaces can be evolved from experience, making autonomous interface design a source of improvement complementary to advances in models and harnesses. We open-source our code at https://github.com/simular-ai/agent-s.