Skill routed evaluation improves accuracy and diagnostics in search systems
SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search
Information RetrievalArtificial Intelligence
Summary
Evaluating the quality of search results is tricky because people see the results as a whole page, but evaluation involves many changing rules. The authors propose SEEK, which keeps evaluation rules separate and uses them only when relevant to check how good search results are and why they might fail. This system learns from human preferences and can update its knowledge without retraining the whole model. They tested SEEK on a popular short-video platform, showing better accuracy and useful problem insights for improving search.
What this means in practice
- •For search engine developers: Improve search result quality assessment by dynamically selecting relevant evaluation criteria for each query and result list.
- •For content platform operators: Diagnose and attribute failures in search rankings on large-scale short-video apps to refine search experience continuously.
Authors
Zhongxin Huang, Songyang Li, Renzhe Zhou, Feiran Zhu, Chenglei Dai, Zhen Xiao, Xuanping Li, Jingwei Zhuo
Abstract
Search quality evaluation provides essential supervision and diagnostic signals for the development and iteration of industrial search systems. Although large language models (LLMs) offer a scalable alternative to manual assessment, reliable automatic evaluation remains challenging: users experience search results at the page level, while the applicable evaluation criteria are multi-dimensional and continuously evolving. Packing all evaluation criteria into a unified prompt introduces irrelevant context and potential criterion interference, whereas internalizing them through post-training tightly couples rule updates with costly model retraining cycles. To address these issues, we propose Skill-routed Evaluation with Evolvable Knowledge (SEEK). Specifically, SEEK externalizes specific search evaluation criteria into a skill bank, dynamically routes relevant skills for each query-result list pair, and employs a task-adapted listwise evaluator to produce page-level judgments and failure mode attribution. A two-stage training pipeline teaches the evaluator to align evaluation criteria with human preferences, while a replay-gated skill bank allows recurring evaluation knowledge gaps to be incorporated without model retraining. Experiments on industrial short-video search show that SEEK improves listwise quality evaluation accuracy and achieves significant progress in attribution diagnosis. SEEK has been deployed at Kuaishou, a short-video platform with over 400 million daily active users, significantly improving the scale and quality of online search evaluation.