Framework identifies causes of search quality problems end to end
WISE: An Agentic Framework for Whole-page Multimodal Search Experience Attribution
Information Retrieval
Summary
Sometimes search engines or recommendation systems work fine but still give poor results, and it's hard to know why. The authors introduce a way to track down where these problems come from inside multi-step systems. They created WISE, a tool that looks at the whole process, from what the user sees back to the computer models making decisions, to find where things go wrong. They also built a new set of test data combining real search results and expert labels to measure how well their tool works.
What this means in practice
- •For search engineers: Detect and trace quality issues in production search systems to improve user experience and model updates.
- •For advertising platform teams: Audit and assure relevance and reliability in multi-step recommendation or ad ranking pipelines by diagnosing error sources.
Authors
Wenyi Xu, Renzhe Zhou, Songyang Li, Feiran Zhu, Ruoqi Zhao, Chenglei Dai, Xuanping Li, Kaiqiao Zhan, Jingwei Zhuo
Abstract
Algorithm-driven services, including search, recommendation, and advertising, rely on multi-stage model pipelines. While research prioritizes model optimization, post-deployment monitoring focuses on operational reliability. Yet operationally healthy systems may still produce poor-quality results, exposing a gap in existing evaluation and monitoring practices. However, a systematic problem formulation and benchmark for this task remain lacking. To bridge this gap, we formulate System Quality Diagnosis (SQD) for algorithm-driven services, a new task that extends quality evaluation beyond identifying poor outcomes to attributing their causes across system components. Focusing on industrial search, we propose WISE, an agentic framework for end-to-end quality diagnosis that discovers user-facing issues and traces their origins to underlying algorithmic components. To improve diagnostic reliability, we introduce ReTrace for constructing evidence-consistent trajectory supervision and Adaptive Harness-Policy Optimization (AHPO) for failure-guided optimization of the agent harness and model policy. To enable systematic evaluation, we construct WISE-BENCH, a real-world multimodal search benchmark that combines user-facing evidence, internal pipeline traces, and expert diagnosis annotations. Extensive experiments show consistent gains across model scales, with WISE achieving 75.20% end-to-end accuracy on Qwen3.5-35B-A3B. In production, WISE supports quality auditing, online quality assurance, and model optimization, reducing diagnosis costs and improving downstream relevance models.