E-commerce relevance improves with verifier guided multi agent reasoning
Learning to Adjudicate: Verifier-Guided Multi-Agent Reasoning for E-Commerce Relevance
Information Retrieval
Summary
Finding the right products in online shops requires understanding what customers want and how products match those needs. The authors show that sometimes different ways of looking at a product can give conflicting answers on whether it fits a customer’s query. They created a system where three expert agents separately analyze the product’s details, the user’s intent, and business rules. A special verifier then evaluates their evidence and guides a debate if they disagree, leading to better final decisions about product relevance. This approach improved search results in tests on a real e-commerce advertising platform.
What this means in practice
- •For e-commerce engineers: Improve product search relevance by integrating multiple expert agents and a verifier to resolve conflicting signals from product, user, and rule analysis.
- •For online advertising teams: Enhance ad targeting and satisfaction metrics by adopting verifier-guided multi-agent reasoning to better judge product-query matches.$Commercial implications: Enables creation of more precise ad relevance models sold to advertisers for better campaign outcomes.
Authors
Shi Shu, Hao Chen, Baopu Qiu, Chao Wei, Weiru Zhang, Xiu Li, Xiaoyi Zeng
Abstract
Relevance modeling is fundamental to e-commerce search, determining how well retrieved products satisfy the needs expressed in a user's query. Accurate judgments require interpreting user intent, comparing product attributes, and applying business-specific matching criteria. Current methods based on large language models (LLMs) leverage contextual understanding and reasoning, focusing on incorporating domain-specific context and enhancing reasoning performance. Despite these advances, difficult query-product pairs remain challenging when evidence from different perspectives supports conflicting conclusions. A single reasoning pass may fail to reconcile these conflicts and ensure that decisive evidence guides the final judgment. To address this challenge, we propose Collaborative Adjudication with Debate and Replay Experience (CADRE), a framework for evidence-based adjudication in e-commerce relevance assessment. CADRE organizes three expert agents sharing a frozen backbone to analyze product attributes, user intent, and business rules in separate contexts, while a trainable Verifier assesses their supporting evidence against the applicable matching criteria to determine relevance. Under expert disagreement, a Verifier-guided debate elicits targeted clarification and revision of disputed interpretations before final adjudication. We optimize the Verifier through multi-turn group relative policy optimization and adapt successful-experience replay so that trajectories yielding correct terminal labels can be reused alongside fresh rollouts in subsequent updates. For efficient online serving, we use CADRE as an offline teacher to distill its relevance judgments into a lightweight student model. Offline evaluations and online A/B tests on an e-commerce search advertising platform demonstrate that CADRE improves relevance classification and key business and relevance satisfaction metrics.