Multimodal AI tools used for wider tasks than text alone suggests
From Images to Tasks: Characterizing Multimodal LLM Interactions in the Wild
Human-Computer Interaction
Summary
People use AI tools that can understand both pictures and words in many different ways, but we don’t fully know what they want to do with images. The authors studied over 40,000 real conversations where people shared images with Microsoft Copilot and categorized the tasks users wanted help with. They found that most tasks combined several abilities like seeing, thinking, and creating text or code. They also noticed that current AI tests don’t fully cover many practical uses people have for these tools. This work helps understand how people really use multimodal AI, which can guide better evaluations and improvements.
What this means in practice
- •For software developers: Prioritize AI evaluation scenarios to better cover real user needs involving images, text, and code together in productivity tools.
- •For ai product managers: Design multimodal features informed by how users actually combine vision and language tasks across diverse workflows.$Commercial implications: Enables creating AI products that better match customer demand for image-plus-text task handling in applications like office software.
Authors
Jinyi Ye, Scott Counts, Gaurav Verma, Kate Lytvynets, Weiwei Yang
Abstract
Multimodal large language models (LLMs) increasingly integrate vision and text, yet how people use them in natural settings remains underexplored. We seek to answer the question: when users upload images, what tasks are they trying to accomplish? Analyzing over 40,000 de-identified image-upload conversations from Microsoft Copilot, we characterize real-world multimodal use through a hierarchical framework of ten capabilities, spanning perception, cognition, and generation. First, we characterize the distribution and composition of these capabilities, finding that the majority of image-upload tasks involve multiple capabilities. Second, we find that multimodal use spans a broader and more diverse task space than text-only interactions, with asymmetric coverage and task classes that rely on cross-modal grounding. Third, mapping observed capability demand onto 253 existing benchmarks reveals uneven alignment between benchmark coverage and real-world use: benchmarks concentrate on perception and reasoning toward fixed answers, while common workflows involving text, code, and data generation remain comparatively undertested. We validate our taxonomy and findings on an independent ChatGPT dataset. Our results provide a large-scale empirical characterization of what users seek to accomplish with multimodal LLMs and highlight opportunities for benchmark design grounded in observed user demand.