Beyond Ranking Accuracy: Evaluating LLM-Cited Feature Rationales for Next Basket Repurchase Recommendation
2026-08-31 • Information Retrieval
Information RetrievalArtificial Intelligence
AI summaryⓘ
The authors studied how well large language models (LLMs) can recommend items a customer might want to buy again, based on past purchases. They found that LLMs are not as accurate at ranking these items as specialized supervised models. However, LLMs can help provide understandable explanations for why certain items are recommended, using human-readable features. This role is more about giving useful reasons than making the best predictions, and their effectiveness varies depending on the dataset. Overall, the authors suggest using LLMs mainly to explain recommendations, not to replace existing ranking methods.
Next-basket recommendationLarge language models (LLMs)Ranking taskRepurchase predictionFeature attributionBehavioral signalsFeature maskingSupervised rankersInterpretabilityRecommendation explanations
Authors
Yanan Cao, Anay Dombe, Murali Mohana Krishna Dandu, Shreeranjani Srirangamsridharan, Sinduja Subramaniam, Yogananth Mahalingam, Evren Korpeoglu, Kannan Achan
Abstract
Next-basket repurchase recommendation is commonly formulated as a ranking task: given a customer's purchase history, the system ranks previously purchased items that may be needed again. In production settings, however, ranking accuracy is only one component of recommendation quality. Customers may also benefit from concise evidence about why an item is recommended now. Large language models (LLMs) offer a potential way to surface such evidence through feature-based, human-readable rationales grounded in interpretable behavioral signals. We construct repurchase features spanning cadence, frequency, recency, user behavior, and item popularity, and evaluate LLMs on two public grocery datasets and one proprietary retail dataset. We investigate (1) whether off-the-shelf LLMs can use these features as next-basket scorers relative to heuristic and supervised rankers, and (2) whether LLM-cited features carry outcome-grounded ranking signal. For the latter, we compare LLM-cited features with model-specific attribution methods under a cross-model feature-masking protocol that measures ranking degradation after masking selected features. Our results show that LLM scores are not competitive with supervised rankers, suggesting that off-the-shelf LLMs should not be used as standalone repurchase recommenders. However, changes in prompt and evidence representation can improve outcome-grounded feature-masking results in some settings even when ranking performance does not improve; the effect is dataset-dependent and does not consistently match attribution baselines. These findings suggest a practical role for LLMs as validated explanation components rather than primary rankers, with rationale quality evaluated separately from ranking accuracy.