LLM agents improve recommendation systems mainly by tuning settings
AutoRSE: Evaluating LLM Agents as Recommendation Engineers for Self-Evolving Recommender Systems
Information Retrieval
Summary
Improving recommendation algorithms automatically is a big challenge. The authors study AI agents that use large language models (LLMs) to do this by themselves. They find these agents help a lot by optimizing things like training settings and combining models, but they rarely come up with new algorithm ideas on their own. When given new ideas from experts, these agents perform even better, showing that generating creative ideas is still a hard problem for them.
What this means in practice
- •For recommendation system engineers: Use LLM agents to automatically tune and ensemble models, improving recommendation performance without manual intervention.
- •For ai platform developers: Integrate idea injection mechanisms to augment LLM agents with expert knowledge for better automatic algorithm innovation.
Authors
Rui Tian, Yi Fang, Yejin Hua, Du Wang, Yuzhe Wang, Minxuan Shan, Zirui Ren, Xinyu Lin, Chongming Gao, Wenjie Wang, Fuli Feng
Abstract
Recommender Systems Engineer Agents (RSE-Agents), which use LLM-based agents to autonomously improve recommendation algorithms, are emerging as a promising paradigm. However, their capabilities and limitations remain poorly understood due to the lack of a unified, standardized evaluation framework and systematic trajectory-level analysis. To address this gap, we introduce AutoRSE, an evaluation framework spanning six representative industrial-grade recommendation tasks, and conduct a large-scale empirical study of three frontier LLMs across three widely used agent harnesses. Our results show that RSE-Agents consistently enhance the performance of backbones on five of the six tasks, with an average relative improvement of 16.1%, indicating that they can already autonomously improve recommendation performance. However, trajectory-level analysis also reveals that current RSE-Agents function primarily as engineering optimizers rather than algorithmic improvers: their gains are largely driven by hyperparameter tuning, training strategies, and model ensembling, while genuine algorithmic innovations are rare. Since the largest gains in recommender systems often stem from algorithmic advances, this behavior underscores a fundamental limitation of current RSE-Agents. To probe this limitation, we perform idea-injection experiments and show that agents equipped with innovative ideas from frontier research or human experts achieve substantially larger improvements than those relying on self-generated ideas. These results suggest that idea generation is a central bottleneck of current RSE-Agents, highlights a key direction for the future recommendation system research. Our implementation is publicly available on GitHub: https://github.com/Ray-Tiann/AutoRSE.