Large study reveals how people use multimodal AI task helpers
Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution
Human-Computer InteractionArtificial IntelligenceComputation and Language
Summary
It can be hard to understand how people really use AI helpers for complex tasks like cooking or fixing things because studies often happen in limited, controlled places. The authors analyzed data from thousands of real users interacting naturally with a conversational AI that helps through voice, text, pictures, and video. They found patterns in how people talk to the AI, what they want to do, how they behave, and what leads to them feeling satisfied. These insights help improve the design of future AI helpers so they can be easier and more engaging to use.
What this means in practice
- •For interface designers: Create better multimodal conversational systems for guiding users through complex tasks based on real-world user behavior insights.
- •For customer experience teams: Improve user satisfaction of AI task helpers by adapting dialogue interactions according to identified behavioral factors.
Authors
Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães
Abstract
Conversational Task Assistants (CTAs) are multimodal dialogue systems that support users in complex real-world tasks such as cooking and DIY through voice, text, image, and video interactions. Prior user studies have focused on controlled settings, leaving limited understanding of real-world CTA usage at scale. In this work, we present a large-scale study of CTA usage based on thousands of users in-the-wild. Our large-scale real-world data analysis unveils new understandings of (i) user-CTA interaction flows, (ii) user intents, (iii) user conversational traits, and (iv) behavioral factors associated with user satisfaction. Our findings reveal key opportunities for future research in CTAs, particularly in user interaction design and task engagement, concluding with concrete design guidelines.