Transformers improve recommendation by predicting many possible next items
Compote: Collapse-Resistant Multiple Hypothesis Learning For Sequential Recommendation
Machine LearningInformation Retrieval
Summary
Predicting what product a user will want next is tricky because many different items could fit their interests. The authors introduce a method called COMPOTE that helps recommendation models consider multiple different next choices at the same time, instead of guessing just one. This method trains the model to specialize different guesses on different item groups, avoiding confusion. Their math shows this approach finds a good way to split predictions, and tests on Amazon data show it works better than existing methods.
What this means in practice
- •For ecommerce platform engineers: Improve product recommendations by modeling multiple possible next choices from user history, boosting accuracy across diverse catalogs.
- •For streaming service developers: Enhance content suggestion by generating separate prediction paths for varied user interests within sequential interaction data.
Authors
Louis Castres Saint Martin, Clement Calauzenes, Alain Rakotomamonjy, Patrick Gallinari
Abstract
Next-item prediction is inherently ambiguous: a single interaction history often admits multiple plausible continuations across distinct catalogue regions. Standard sequential recommenders compress these alternatives into a single prediction, conflating user intents. We introduce COllapse-resistant Multiple hyPOThesis lEarning (COMPOTE), a multi-hypothesis transformer trained with an annealed objective that transitions from uniform supervision to a winner-takes-all assignment. This mechanism forces hypotheses to specialize into distinct catalogue regions, while inference relies on an efficient, normalizer-free max-similarity rule. Theoretically, we show that in the population case, the optimal solutions partition the conditional next-item distribution according to a max-entropy quantization principle, while pooled scores preserve the Bayes-optimal ranking. We also prove a generalization bound scaling logarithmically with the number of hypotheses. Across three Amazon Reviews 2023 benchmarks, COMPOTE outperforms strong ID-based, generative, and multi-interest baselines using item IDs alone. Matched-capacity ablations confirm that performance gains stem directly from the annealed objective, preventing the head collapse observed in conventional multi-head architectures, rather than added capacity, ensembling, or routing.