Pailitaommsearch improves multimodal ecommerce searches and sales
PailitaoMMSearch2.0 Technical Report: Scaling Generative Retrieval to Unified End-to-End Multimodal Search
Information RetrievalArtificial Intelligence
Summary
Searching for products using both images and text is tricky, especially when the demands and resources vary widely. The authors developed PailitaoMMSearch, a system that understands images and free-form instructions together to find products accurately across different shopping intents and online settings. They improved the system using large-scale data, continual learning, and carefully balanced training techniques. When tested live on Taobao, the system led to more product views, transactions, and total sales.
What this means in practice
- •For e-commerce platform teams: Build multimodal search engines capable of interpreting images and free-form text to improve product discovery and sales conversion.$Commercial implications: Enables scalable, unified multimodal search products that increase transactions and revenue on large online marketplaces.
- •For online retail operations: Adjust search system quality and resource use dynamically to balance performance and cost in diverse operational scenarios.
Authors
Xiaohan Ye, Zihan Gong, Jingqian Zhao, Lianyu Du, Yunmeng Shu, Xu Chen, Zesheng Zhou, Renchun You, Bocheng Chen, Jinsong Lan, Haihong Tang, Bo Zheng
Abstract
Generative retrieval models have been increasingly adopted in search and recommendation systems. However, industrial generative retrieval models are commonly specialized to a fixed online scenario: they learn from scenario-specific interactions, support a narrow set of query forms and retrieval objectives. We study how to remove these couplings and build a unified end-to-end multimodal search system for open-ended image--text queries, heterogeneous shopping intents, and online scenarios with different resource constraints. We present \textbf{PailitaoMMSearch}, which directly generates product Semantic IDs from a raw query image and free-form instruction and scales along three dimensions. \textbf{Data scaling} uses a reusable toolset that can be composed into intent- and lifecycle-specific workflows for cold-start synthesis, quality control, real-traffic refinement, and continuous data production. \textbf{Intent scaling} progressively equips one model with e-commerce product knowledge, multiple retrieval relations, and adjustable reasoning budgets through continual pre-training (CPT), mixed-budget supervised fine-tuning (SFT), and unified reinforcement learning (RL). The RL stage combines intent-balanced optimization and Sample--Beam rollouts aligned with multi-candidate serving. \textbf{Scenario scaling} exposes quality--cost operating points across model capacity and reasoning budget. The system is deployed in multimodal search applications on Taobao Pailitao. A controlled online A/B test reports statistically significant improvements, including item page views $\textbf{+17.90\%}$, transactions $\textbf{+27.04\%}$, and gross merchandise value $\textbf{+8.94\%}$.