RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation
2026-07-10 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors studied how to choose among different recommendation methods to get better suggestions for users while keeping costs in mind. They tested two ways: picking one entire recommendation list per user request or mixing items from multiple recommenders. Their experiments showed that just picking one method per request didn’t work well, but combining recommendations at the item level performed better, especially when including a large language model (LLM) reranker selectively. They conclude that blending recommendations item-by-item is a better approach than choosing one whole list in scenarios with limited options.
recommender systemscollaborative filteringsequential modelscontent-based retrievallarge language modelsagent routingitem-level aggregationMovieLens-1MHR@10NDCG
Authors
Kaiji Zhou, Vladimir Kalmykov, Yue Feng
Abstract
Recommender systems increasingly face a choice among heterogeneous agents -- collaborative filters, sequential models, content-based retrievers, and LLM-based rerankers -- yet no single agent is uniformly best. We study this choice as task-aware agent ranking under cost constraints using RouteRec, a framework that compares request-level hard selection with item-level learned aggregation over four traditional recommender agents and one LLM reranker agent. On MovieLens-1M, the full quality oracle has substantial headroom (HR@10 = 0.584), confirming that useful cross-agent signal exists. Under a leakage-free 5-fold out-of-fold protocol, however, hard selection remains below BM25 (0.223 vs. 0.254), and selective LLM escalation does not improve it. The same protocol yields a different outcome for learned aggregation: its cheap-only variant matches BM25 in HR and has a higher NDCG point estimate (0.123 vs. 0.114), while gated all-agent aggregation reaches HR@10 = 0.295 with 70.2\% LLM calls. The resulting lesson is not that routing is solved, but that request-level selection of one complete agent list is too coarse for this sparse fixed-candidate setting; item-level aggregation is the more promising action space.