Personalised search improves ranking by balancing behaviour and meaning
Robust Fusion of Semantic and Behavioural Signals for LLM Reranking in Personalised Search
Information Retrieval
Summary
Search engines that learn from user history can do better by understanding what people mean and what they like. The authors found that simply adding past user behavior into search ranking can make the system too reliant on old data and less able to handle new or rare searches. They developed a training method that sometimes includes this behavior data and sometimes leaves it out, helping the system stay strong even when such data is missing. Tests on a music streaming platform showed better search results both offline and live with their approach.
What this means in practice
- •For search engine developers: Improve personalised search ranking by training models to handle user behavior signals optionally without losing understanding of query meaning.
- •For music streaming platform engineers: Increase relevance of search results in music apps by integrating user interaction statistics with semantic understanding in reranking models.
Tested on one dataset.
Authors
Aleksandr V. Petrov, Nathan Stein, Erik Lybecker, Emma Schüldt, Daniel Lazarovski, Hugues Bouchard, Mounia Lalmas
Abstract
Personalised search must satisfy query intent while incorporating user context and historical interactions. LLM-based cross-encoders provide a single reranking interface, but injecting predictive behavioural statistics into their prompts can encourage shortcut learning: reliance on historical signals at the expense of semantic and user-context patterns that generalise to sparse or unseen searches. We study this problem in the personalised search system of a large-scale audio streaming platform using Query Slice Stats (QSS), an interaction-derived behavioural feature summarising historical success for query-candidate pairs. Naive QSS injection improves ranking when the feature is available but reduces robustness when it is removed. We address this with deterministic dual-sample feature-dropout training, which presents each example once with QSS included and once with QSS removed. Offline, QSS injection improves ranking quality by 13.3% when available. Dual-sample training preserves these gains while improving performance under QSS-removed evaluation by 4.0% relative to naive QSS training. In a live online test, both QSS-aware variants improve search success by roughly 2%. The aggregate test does not distinguish dual-sample from features-only training; the cold-start comparison is directionally consistent with the offline results. Paired feature-present and feature-removed training can therefore reduce the tension between exploiting strong behavioural statistics and remaining robust when they are unavailable.