Persona prompts improve query understanding but increase hallucinations
Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search
Computation and LanguageInformation Retrieval
Summary
Search systems often need lots of example questions to understand what users want, especially for new shopping sites. The authors tested ways to create fake example questions using different personality prompts to help the system learn. They found that some types of personality prompts helped the system get better at understanding questions, while others made it produce unrealistic or 'hallucinated' questions. They also developed ways to measure how close the fake questions are to real ones without needing extra labels. This research helps build better search tools by carefully choosing how to create training data.
What this means in practice
- •For e-commerce search engineers: Improve new product search systems by using persona-based prompts to generate better training queries that align with real customer searches.
- •For chatbot developers: Use personality-conditioned query generation to create diverse but realistic user inputs for training conversational agents in retail domains.
Authors
Filippos Ventirozos, Tom Armitage, Mahmoud Oshagh, Peter A. Appleby, Matthew Shardlow
Abstract
Synthetic supervision can bootstrap semantic parsers in new search domains, but prompt-induced diversity is useful only when generations are faithful to the structured intent. We explore this for cold-start e-commerce search: 118,098 synthetic pairs produced by 6 LLMs with 9 persona-conditioned prompt families on a 47-field UK automotive schema, against 1,214 human-annotated and 15,304 production queries. We show that the IPIP personality bundle improves parser F1 in all four matched contrasts (Holm-corrected $p<.001$), and all three bundles have a positive main effect over the complete $2^3$ grid. Syntax conditioning is the axis that moves generations towards real traffic. It also induces lexical diversity. Alignment sKL, the divergence between von Mises-Fisher distributions fitted to synthetic and human-annotated queries, is an unlabelled proxy for parser F1 ($ρ=-0.917$ against human queries written for simulated shopper scenarios, lower is closer). An LLM committee, checked against a blinded author pass, triages the 927 queries missed by every parser and labels 37% as hallucinated. Dropped constraints dominate. The four syntax-bearing families have the largest hallucinated shares, 38 to 42%, while IPIP alone has the lowest at 28%.