Hybrid gpu and cpu system improves personalized search at trillion scale
Hybrid GPU-CPU Retrieval for Personalized Search at Ultra-Large Scale
Information RetrievalDistributed, Parallel, and Cluster ComputingMachine LearningPerformance
Summary
Searching billions to trillions of documents to find what each user wants is very hard because powerful methods require a lot of memory but must respond quickly. The authors describe a system that uses both GPUs and CPUs together: GPUs do deep and detailed searching on a smaller set, while CPUs quickly scan a much larger set with simpler methods. This mix helps balance speed, resource use, and personalization, and it runs in production with better results than previous CPU-only approaches.
What this means in practice
- •For search infrastructure teams: Run a combined GPU-CPU retrieval system that balances detailed personalization and large inventory coverage under latency constraints at ultra-large scale.
- •For cloud service providers: Plan and assign GPU and CPU resources economically for personalized search workloads to optimize model depth and browsing breadth.
Authors
Hao Fu, Jichao Sun, Baiting Zhu, Qiaoling Liu, Yan Shi, Cheng Lu, Liu Liu, Yubo Wang, Xin Yao, Xiangyu Niu, Xu Dong, Wenhan Lyu, Chiyao Shen, Yinjie Huang, Minglei Chen, Shuai Ding, Li Fan, Xiao Kong
Abstract
Embedding-based retrieval on user-generated content at the trillion-document scale exposes a sharp conflict between two production demands: deep, expressive personalization for queries with rich user intent, and broad coverage of a massive inventory under fixed latency and resource budgets. We characterize this as the personalization-scale paradox: hosting the full serving inventory in GPU memory is too resource intensive, while CPU compute cannot execute the same interaction-heavy model on the latency-critical path. We present a hybrid GPU-CPU co-serving system that resolves the paradox through orchestration rather than a new model class. A high-depth GPU pathway fuses retrieval and interaction pre-ranking over a curated online pool on the order of a billion documents, while a high-breadth CPU pathway searches an independently selected online inventory roughly twenty times larger with lightweight personalized scoring. Either or both pathways can run per request; candidates are deduplicated before shared downstream ranking. The system is deployed in production. A full-system A/B test against the legacy CPU-only configuration improves model-scored relevance and substantive engagement, while separate pathway experiments show positive value at their own deployment scopes. Retrieval logs show that the pathways contribute structurally distinct candidates, production serving measurements characterize their latency, and a matched capacity plan quantifies the economic rationale for assigning modeling depth to GPUs and inventory breadth to CPUs. Together, these results validate a practical, independently evolvable depth-breadth architecture for ultra-large-scale personalized search.