Tool orchestration challenges slow progress in smart recommendation systems
RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent
Information Retrieval
Summary
Making recommendation systems that can understand unclear user requests and use multiple tools well is hard. The authors created RecToolBench, a test that checks how well recommendation agents handle fuzzy user instructions using many tools in different ways. They found that even if the tools are used correctly, the systems still struggle to give good recommendations, especially when tasks need several steps. This shows that handling unclear user needs and coordinating tools is a big challenge for smarter recommendation programs.
What this means in practice
- •For software engineers: Test and improve recommendation agents that must interpret vague user requests and coordinate multiple tools effectively to generate better results.
- •For product managers: Evaluate candidate recommendation platforms for their ability to handle complex user needs involving multiple external tools before product integration.
Authors
Xiao Chen, Yicheng Zhao, Yingying Wu, Zhendong Chu, Changyi Ma, Qingsong Wen, Xuan Song
Abstract
Recent advances in agentic recommender systems are shifting recommender systems from passive filtering engines to instruction-following agents that use external tools to resolve user intent. However, existing benchmarks often assume explicit user intent, simplified tool environments, or isolated function calls, leaving realistic tool orchestration for recommendation underexplored. To bridge this gap, we propose RecToolBench, a Model Context Protocol (MCP)-based benchmark for evaluating tool-using recommender agents under fuzzy user instructions. RecToolBench contains more than 1,200 executable tasks across three recommendation domains, 13 MCP servers, and 32 tools, spanning single-tool calls, parallel tool calls, sequential tool chains, and hybrid tool orchestration. We construct RecToolBench with a scalable synthesize--fuzzify--judge pipeline that generates executable fuzzy recommendation tasks, and evaluates agent trajectories using rule-based execution checks and rubric-based LLM evaluation. Experiments on representative LLMs show that syntactically valid tool calls do not guarantee successful recommendations. Models struggle with semantic parameter grounding, multi-step evidence integration, and grounded final recommendations, especially as orchestration complexity increases. Our results identify tool orchestration under fuzzy user intent as a major bottleneck for agentic recommender systems. Our data and code are available at https://github.com/ShawnChenn/RecToolBench.