Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
2026-06-29 • Information Retrieval
Information RetrievalMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) perform when used to rerank recommendations in systems across five different areas. They found that while LLM rerankers have some semantic understanding, they generally do not outperform traditional methods, especially when the target item isn’t guaranteed to be in the candidate set. They propose a new method called LHF that combines results from multiple retrievers to improve item coverage, but LLM reranking still struggles to utilize this improved retrieval fully. Overall, the authors show that current ways of using LLMs in recommendation pipelines have limitations, particularly for brand-new items with no prior data.
large language modelsrerankingrecommendation systemscold-start problemretrieval coveragemulti-retriever fusioncollaborative filteringcontent-based filteringlearned hybrid fusionoracle coverage
Authors
Zhe Dong, Fang Qin, Manish Shah, Yicheng Wang
Abstract
Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes. We test this assumption with a five-domain benchmark that explicitly separates reranking quality from retrieval coverage. In a positive-controlled regime where the gold item is guaranteed present, calibrated LLM rerankers fail to consistently outperform strong collaborative and content baselines under natural traffic, and within-family scaling from Qwen3-8B to Qwen3-32B narrows but does not close the gap on most domains. In a retrieval-realistic regime where the gold item is not injected, the bottleneck is more severe: standard single retrievers place the gold item in a 200-item pool only 4.6-22.9% of the time, largely because 32-91% of cold-start targets are brand-new items with no training interactions. We introduce LHF, a validation-trained learned hybrid fusion layer over a multi-retriever union pool, as a retrieval-side realizability baseline. LHF is the only combiner we test that beats every single retriever on all five domains and recovers 17-61% of oracle coverage headroom on content-rich domains, but only 5-7% on collaboratively strong domains. End-to-end experiments reveal the remaining mismatch: learned non-LLM ranking exploits the LHF pool, while prompt-level LLM reranking often degrades it. LLMs exhibit pockets of semantic cold-start advantage, especially in text-rich domains when the item is already present, but this advantage is largely unreachable in current retrieve-then-rerank pipelines. We release the benchmark protocol, splits, prompts, evaluation tooling, and archived reproducibility artifacts: data at https://doi.org/10.5281/zenodo.20991039 and code at https://doi.org/10.5281/zenodo.20993306.