AI moderation matches humans and finds more customer needs
AI-Moderated Interviews for Market Research and Digital Twins Calibration
Computers and SocietyArtificial IntelligenceHuman-Computer InteractionMultiagent Systems
Summary
Finding out what customers want is important but expensive when done by humans. The authors tested AI-led interviews against human and simple surveys with real participants. They found that AI interviews gathered as many insights as human ones and even discovered more customer needs when the budget was the same. However, people felt more emotionally connected talking to humans. The AI data helped make digital models of customers, which predicted responses better than simple demographic profiles but not better than static surveys.
What this means in practice
- •For market researchers: Use AI-moderated interviews to discover more customer needs within a fixed budget than traditional human interviews.
- •For marketing analytics teams: Create digital twins from AI interviews to better predict consumer responses compared to basic demographics-based personas.
Authors
Yuting Deng, Jingxuan Liu, Olivier Toubia, Naman Jain
Abstract
AI-moderated interviews are emerging as a scalable market-research method for generating consumer insights and building consumer "digital twins." Yet it remains unclear whether they match human-moderated interviews or improve on simpler, static data collection methods. In a pre-registered, between-subjects study (N = 317) with three industry partners, we compare AI-moderated (N = 139), human-moderated (N = 24), and static interviews (N = 154). AI moderation matches human moderation in depth, covers more themes, and, holding budget constant, recovers significantly more customer needs than human moderation or static interviews. However, participants sound more emotionally engaged when speaking to a live human. We then create digital twins using interview data and evaluate each twin against the participant's own held-out responses to six real-world marketing stimuli. We find that digital twins created from AI-moderated interviews predict consumer responses better than demographics-only personas. However, the additional richness from AI moderation does not translate into better quantitative predictions compared to static interviews. By analyzing open-ended thoughts generated from humans versus their twins, we find that prediction errors are connected both to differences in (self-reported) thinking styles between twins and humans, and to gaps between training and validation data (i.e., asking questions that are too far out of distribution).