Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation
2026-07-20 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how people shop online using not just words, but also images and mixed instructions. They found current search methods either focus on one type of input or are too general to understand specific shopping needs. To fix this, they created Pailitao-MMSearch, a special model combining text and images tailored for e-commerce. Their model showed better sales results on Taobao, proving it understands products and user intents better than previous systems.
e-commercemultimodal searchvision-language modelsproduct retrievalnatural language processingimage searchcontinual pre-traininglarge language modelsTaobaoGross Merchandise Volume (GMV)
Authors
Xiaohan Ye, Xu Chen, Zihan Gong, Jian Ding, Lianyu Du, Baicheng Chen, Yunmeng Shu, Jingqian Zhao, Zhixiang Zhao, Shuaiqi Jia, Chong Ma, Shuwen Xiao, Xiangheng Kong, Yuan Gao, Jun Song, Jinsong Lan, Xiaoyong Zhu, Bo Zheng
Abstract
The evolution of e-commerce has fundamentally transformed how users search for products, shifting from simple text-based keyword queries to complex multimodal interactions that seamlessly combine product images, natural language descriptions, and mixed-intent instructions. However, existing approaches face a critical dilemma: single-modal specialist models, deployed independently for text retrieval, visual search, and voice recognition, operate in isolation and cannot handle cross-modal queries, while general-purpose vision-language models lack the domain-specific knowledge necessary for fine-grained product understanding, user behavior modeling, and commercial intent reasoning. In this work, we present Pailitao-MMSearch, one native e-commerce multimodal search foundation model designed to bridge this gap. Our approach introduces three key innovations: (1)HybSID (Hybrid Semantic ID);(2)a two-stage continual pre-training strategy; and (3)a hybrid reasoning post-training pipeline. Built upon Qwen and deployed on Taobao's Pailitao multimodal search platform, Pailitao-MMSearch achieves substantial improvements in online A/B testing, including up to +13.61\% in Gross Merchandise Volume (GMV) and +8.21\% in transaction volume compared to traditional multi-modal search pipeline, demonstrating the effectiveness of our native e-commerce multimodal search large language models.