Generative Proxy: Synthesizing Proxy-Based Interfaces for Real-World Interaction Across AR Glasses
2026-08-10 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors focus on making it easier to interact with real objects in augmented reality (AR), especially on small AR glasses that don’t support advanced features like hand tracking. They created a method called Generative Proxy that automatically designs simplified virtual versions (proxies) of real objects for interaction based on the scene, user goal, and device features. This method chooses good proxy designs that fit the device and task and has been tested in different scenarios with positive expert feedback. Their work shows a new way to interact with AR that could work well on future AR glasses.
augmented realityproxy interfaceuser interface generationAR glassesinteraction designconstrained synthesisdirect manipulation theorydevice capabilitiessemantic distancearticulatory distance
Authors
Xianhao Carton Liu, William Chastek, Eric J Gonzalez, Mar Gonzalez-Franco, Chen Zhu-Tian
Abstract
Interacting with real-world objects in AR is difficult, especially when targets are distant, cluttered, or occluded. These challenges are amplified on emerging lightweight AR glasses, which often lack binocular or large field of view on display, but also continuous inputs, such as hand or eye tracking. Proxy-based interfaces offer an alternative by allowing users to interact with virtual abstractions of physical objects that can be repositioned, reorganized, and adapted to the task and device. However, designing such interfaces is currently manual and highly device-specific. We present Generative Proxy, a method for automatically generating proxy-based interfaces from three specifications: scene, intent, and device capabilities. We formulate generation as a constrained synthesis problem that first produces valid interfaces for the target device and task, then ranks candidates using semantic and articulatory distance inspired by direct manipulation theory. We demonstrate Generative Proxy across diverse scenes, device profiles, and user intents. Expert evaluation shows initial evidence that generated proxy UIs are useful and usable, highlighting proxy-based abstraction as a promising interaction paradigm for future AR glasses.