Computer-use agents interact visually to fix and test software changes
CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering
Artificial IntelligenceSoftware Engineering
Summary
Software developers not only write code but also use and observe software running with its graphical interface to find and fix problems. The authors created CUA-SWE, a new testing setup where AI agents must look at a program's interface, understand what the software is doing, and change the code accordingly to fix issues. The system checks if the changes work based on specific tests connected to what the agents see and do. This helps study how AI can better work with visual feedback in software development to complete tasks that need both coding and using the software.
What this means in practice
- •For software developers: Test AI tools that fix software bugs by examining the program’s running interface and updating code accordingly in integrated tasks.
- •For automation engineers: Build automated systems that combine code editing with visual checks to verify software changes across multiple domains.
Authors
Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, Aojie Yuan, Xiaolin Zhou, Haiyue Zhang, Yue Zhao, Xiyang Hu, Shuli Jiang
Abstract
Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works. Existing coding agents and computer-use agents are largely studied in isolation, leaving this integrated development process underexplored. Diagnosing a runtime interaction failure requires agents to connect visual observations with the responsible code, then use the application again to verify the repair. We introduce CUA-SWE, a benchmark, environment, and evaluation pipeline for software engineering with computer use. Beyond studying how GUI feedback supports diagnosis and repair, we ask whether agents can complete software engineering tasks when required specification or operational information is available only through the running application's visual interface. CUA-SWE spans four software engineering domains and requires agents to modify code and configuration, execute commands, interact with running software, and inspect visual feedback within the same task. Each task includes deterministic, task-specific tests that verify whether the resulting software satisfies the requirements and preserves specified behavior. Our evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes. We examine performance across domains and task information requirements, alongside the development behaviors associated with successful repairs. CUA-SWE provides a unified testbed for studying how agents use visual feedback and interaction to guide software engineering, with executable correctness criteria for the resulting software.