PersonalQuery-Benchmark measures query variation effects on search results
PersonalQuery-Benchmark:Evaluating Retrieval Robustness to Personalized User Expression Differences
Information Retrieval
Summary
People often describe the same information need in different ways when searching, but most search tools don't consider how these personal ways of phrasing affect results. The authors created a new test called PersonalQuery-Benchmark that includes examples of various styles and mistakes different users make. They showed that even when the question is the same, how it’s expressed changes which answers appear and that different search methods react differently to this variation. This benchmark helps to study and improve how search systems handle personal writing styles.
What this means in practice
- •For search engine developers: Evaluate and improve retrieval systems' robustness to different user query expressions using realistic personalized query variations.
- •For e-commerce platform teams: Enhance product search results by testing how algorithms handle varied customer query styles and writing errors across Amazon product domains.
Authors
Wenyu Liang, Hongzhi Yin, Tongtong Wu, Fan Liu, Kexin Dai, Weiqing Wang
Abstract
Existing personalized information retrieval methods mainly model differences in user needs through user preferences and query intent, while paying little attention to the personalized expression differences that arise when different users express the same retrieval need. Existing real query data typically lack stable user identities, long-term expression histories, and intent-matched counterparts, making it difficult to systematically evaluate the impact of personalized expression differences on retrieval outcomes. To this end, we propose PersonalQuery-Benchmark (PQB). PQB models personalized syntactic expression and personalized writing errors from users' historical texts, constructs personalized syntactic expression queries and personalized writing-error queries, and validates benchmark quality through LLM assessment and independent human evaluation. PQB covers three Amazon product domains with 6,516 query instances in total, and is evaluated on five retrieval models representing lexical, learned sparse, dense bi-encoder, late-interaction, and hybrid architectures, together with two LLM-based ranking models. Results show that even when the information need is held fixed, personalized expression differences still induce measurable ranking variation; different retrieval models exhibit clearly different sensitivities to syntactic variation and writing-error variation, while higher overall retrieval effectiveness does not necessarily correspond to stronger robustness against expression variation. PQB provides a new benchmark and evaluation framework for systematically studying personalized expression robustness. The associated code and dataset are available at the following address: https://github.com/cpxjazz123/PersonalQuery-Benchmark.