Mixed reality interfaces adapt with speech and touch combined
Say It or Grab It? Conversational Control of UI Adaptations in Mixed Reality
Human-Computer Interaction
Summary
Mixed reality devices like augmented reality glasses need to change their display and controls as a person works or moves around. This study asked users how they might say what changes they want and built a system called ConverseMR that listens to spoken requests to adjust the interface automatically. They found that talking to the system helps with broad and flexible changes, while touching controls works better for precise adjustments. Combining speech and touch gave users the best experience for controlling their mixed reality environments.
What this means in practice
- •For mixed reality developers: Create adaptive interfaces that users can adjust precisely by touch or broadly via speech commands in changing environments.
- •For assistive technology teams: Enable users with limited manual dexterity to control mixed reality interfaces through spoken commands that continuously adjust to context.
Authors
Sina Elahimanesh, Joao Marcelo Evangelista Belo, Isidora Jeknic, Alexander Koller, Anna Maria Feit
Abstract
Mixed reality interfaces must continuously adapt to a user's changing task and environment, yet adaptation policies are fixed in advance, either preset by the system or configured by the user. Natural language offers a way to state changing needs mid-task, but whether users can express interface behavior this way, and what it gains over direct manipulation, is unknown. We first elicited how users verbalize adaptation needs (N=19). We then built ConverseMR, which translates a spoken request into adaptation objectives and constraints and satisfies them continuously as the user's context changes. In a within-subjects study (N=24), conversational and direct control proved complementary: direct manipulation for precision, ConverseMR for reach and expressive breadth, with their combination rated highest. Spoken commands were rarely final and users frequently refined and replaced them upon experiencing the result, characteristic of, what we call, user-led adaptation, where users communicate adaption needs and the system identifies a solution.