Generative recommenders speed up item suggestions by splitting thinking from spelling
Separating Thinking from Spelling for Efficient Generative Recommendation
Information Retrieval
Summary
Making item recommendations on big web platforms is often slow because generating each item's identifier takes a lot of steps. The authors propose a method called Deco that speeds up this process by separating how the system understands what to recommend from how it writes down the item IDs. Deco uses a smaller, faster decoder to handle the details after the main system summarizes the user's history. This approach keeps accuracy high while making the whole recommendation process up to 8.6 times faster.
What this means in practice
- •For e-commerce teams: Increase recommendation throughput and reduce latency when generating personalized product suggestions under tight time limits.
- •For music streaming services: Generate larger pools of recommended songs quickly while maintaining or improving suggestion quality in real-time user interfaces.
Authors
Danil Gusak, Anna Volodkevich, Ivan Oseledets, Evgeny Frolov
Abstract
Generative recommenders retrieve items by writing their semantic IDs (SIDs) code by code under trie-constrained beam search, an approach now deployed on large web platforms. Standard autoregressive interfaces pay for the SID length K twice: a history of T items enters the backbone as TK code tokens, and every beam-search step runs through the backbone, so returning B candidates costs O(BK) backbone decode tokens per user. The second cost often dominates, and it grows with the backbone, the SID depth and the candidate budget. We introduce Deco (decoding from concepts), which separates reading the history from writing the identifier. The backbone reads each history item as one fused token and condenses the history once into a compact next-item concept; a light KV-cached decoder conditioned on the concept and the generated prefix runs the entire beam search, while the identifiers, the trie and the autoregressive factorization stay unchanged. The decoder approaches full-backbone accuracy with cross-entropy alone, and distillation improves it further. On seven datasets from e-commerce, movies and music streaming, Deco achieves comparable or better accuracy than both a standard generative recommender and the full-backbone teacher with fused inputs, while delivering 3.2--8.6$\times$ their throughput under a 50 ms p95 latency budget in a production-style serving pipeline. Delegating beam search makes larger candidate pools and models practical: Deco generates 1000 candidates faster than its teacher generates 100, and with a 1.5B-parameter LLM backbone, it matches the 69M-parameter teacher's throughput while achieving higher accuracy.