Interface Quilt creates patches to change app interfaces without code
Interface Quilt: Modifying Applications with Computer Use Agents and Generative UI
Human-Computer Interaction
Summary
Changing how an app looks or works is often hard because apps usually don’t let you directly change their code or settings. The authors show how computer-use agents—smart tools that understand app interfaces—and AI systems that can create new interfaces can be combined to patch apps from the outside. Their Interface Quilt system creates new interface layers on top of existing apps that can change their buttons, look, and features without needing the app’s original code or APIs. This approach works on many different apps and lets people modify software more easily.
What this means in practice
- •For software integrators: Modify legacy or closed-source applications to add or change features without needing source code access.
- •For call center operators: Adapt existing customer support software interfaces for better usability or added controls without vendor cooperation.
Authors
Leping Qiu, Jo Vermeulen, Tim Zindulka, George Fitzmaurice, Fraser Anderson
Abstract
Modifying an existing application is difficult when it does not provide programmatic access for that change. Recent advances in computer-use agents (CUAs) enable richer understanding and operation of existing interfaces, while generative UI (GenUI) makes it practical to generate new ones. We examine how these advances expand software modification through a design space of what can be changed and a framework for how modifications can be implemented. We then introduce Interface Quilt, a system that combines CUAs and GenUI to generate interface patches, which are new interfaces displayed over an existing application to modify its presentation, interaction, and capabilities. Through demonstrations across a variety of applications, we show that interface patches support a range of modifications without source code, APIs, or accessibility data.