WeChat improves search accuracy and efficiency with new generative retrieval system
WeGR: Unified Generative Retrieval for Industrial Search Systems in WeChat
Information RetrievalArtificial Intelligence
Summary
Searching in big platforms like WeChat can be slow and inaccurate when trying to find the right item. The authors created a new system called WeGR that improves how the search engine understands and ranks results by learning better ways to represent and gather information about items. Their approach uses special techniques to handle the way search queries and items relate while also improving learning during system training. Testing showed this method not only finds better matches but also uses less computing power, helping millions of WeChat users find what they want more quickly and reliably.
What this means in practice
- •For e-commerce platform engineers: Build efficient search engines that deliver more relevant product results with less computing power for large-scale shopping platforms.
- •For social media app developers: Improve in-app item or content search accuracy and user engagement by integrating collaborative signals and advanced training for retrieval systems.
Authors
Jiajun Han, Guoxin Ma, Felix, Liangsheng Xu, Ruiqi Yu, Yingdi Guo, Zongbing Wang, Xinhua Ye, Chenyan Hua, Xin Xie, Pengcheng Yang, Haijun Wu
Abstract
Generative retrieval (GR) offers a promising paradigm for industrial search by unifying candidate generation within an autoregressive framework. However, its effectiveness is limited by two key challenges. First, existing Semantic ID (SID) tokenizers struggle to incorporate collaborative signals while maintaining effective codebook utilization, limiting their ability to distinguish items and achieve high-quality retrieval. Second, existing reinforcement learning (RL) methods suffer from severe reward sparsity under exact SID matching, leaving many sampled trajectories without effective learning signals and hindering preference optimization. To address these challenges, we propose WeGR, a unified generative retrieval framework consisting of two complementary components. Specifically, the Collaborative and Collapse-free Tokenizer (CoCo-VAE) incorporates query-to-item and item-to-item collaborative signals while mitigating codebook collapse through codebook refresh and complementary regularization, enabling more informative and balanced SID representations. Building upon these SIDs, Dynamic Horizon Policy Optimization (DHPO) progressively expands both reward evaluation and policy optimization horizons, reducing learning difficulty while leveraging behavioral reward signals at different levels of the SID hierarchy, without relying on auxiliary reward models. Extensive offline experiments demonstrate the effectiveness of WeGR in improving retrieval and ranking performance. Moreover, a two-month online A/B test in WeChat E-commerce Search demonstrates relative improvements of 2.539% in orders, 1.508% in click-through rate, and 1.381% in conversion rate, while reducing online CPU and GPU usage by 25.5% and 26.1%, respectively. The system has been successfully deployed across multiple search scenarios in WeChat, serving billions of users, generating tens of millions of PVs daily.