MAESTRO: Adapting GUIs and Guiding Navigation with User Preferences in Conversational Agents with GUIs
2026-04-07 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors created MAESTRO, a chatbot that helps users with tasks like booking movies by better understanding and using their preferences throughout the process. Unlike older chatbots that just follow steps, MAESTRO remembers user likes and dislikes and adjusts the on-screen options to fit these preferences. It also detects when choices conflict and suggests going back to fix problems instead of forcing users to start over. The authors tested MAESTRO with 33 people and compared it to a simpler chatbot to see how well it worked.
Task-oriented chatbotGraphical User Interface (GUI)Preference memoryNatural language understandingWorkflow navigationBacktrackingConversational Agent with GUI (CAG)In-place operatorsUser preferencesMulti-step tasks
Authors
Sangwook Lee, Sang Won Lee, Adnan Abbas, Young-Ho Kim, Yan Chen
Abstract
Modern task-oriented chatbots present GUI elements alongside natural-language dialogue, yet the agent's role has largely been limited to interpreting natural-language input as GUI actions and following a linear workflow. In preference-driven, multi-step tasks such as booking a flight or reserving a restaurant, earlier choices constrain later options and may force users to restart from scratch. User preferences serve as the key criteria for these decisions, yet existing agents do not systematically leverage them. We present MAESTRO, which extends the agent's role from execution to decision support. MAESTRO maintains a shared preference memory that extracts preferences from natural-language utterances with their strength, and provides two mechanisms. Preference-Grounded GUI Adaptation applies in-place operators (augment, sort, filter, and highlight) to the existing GUI according to preference strength, supporting within-stage comparison. Preference-Guided Workflow Navigation detects conflicts between preferences and available options, proposes backtracking, and records failed paths to avoid revisiting dead ends. We evaluated MAESTRO in a movie-booking Conversational Agent with GUI (CAG) through a within-subjects study with two conditions (Baseline vs. MAESTRO) and two modes (Text vs. Voice), with N = 33 participants.