Semantic id design shapes but does not fully determine generative recommendation quality
What Makes a Good Semantic ID for Generative Recommendation? A Reproducibility Study
Information Retrieval
Summary
Generating recommendations by representing items with special codes called semantic IDs has become popular, but it’s unclear what makes these codes work best. The authors ran many tests to see how different ways of making these codes affect recommendation results. They found there isn’t one best way — the effectiveness varies by dataset and method. Also, making the codes longer or using bigger models doesn’t always help. Different designs keep different parts of item similarity, so they each have unique strengths.
What this means in practice
- •For recommender system engineers: Choose or combine different semantic ID designs to optimize recommendation quality depending on dataset characteristics.
- •For machine learning platform teams: Improve generative recommendation pipelines by avoiding longer semantic IDs or larger models when they do not improve results.
Authors
Yufei Chen, Junchen Fu, Jujia Zhao, Yukun Zhao, Zhaochun Ren
Abstract
Generative recommendation has emerged as an active research direction, where items are commonly represented by semantic IDs (SIDs): discrete codes generated token by token. Despite strong empirical results, SID designs vary widely in construction strategy, codebook organization, and code length, making their true impact on recommendation performance unclear. We conduct a large-scale reproducibility study to systematically investigate the impact of semantic ID design on generative recommendation under a unified experimental framework. We focus on a fundamental question: What makes a good semantic ID for generative recommendation? To answer this question, we examine four aspects: the relative effectiveness of different semantic ID designs, the connection between codebook utilization and recommendation quality, the effect of semantic code length, and the influence of semantic ID design on local item semantic preservation. Through a unified evaluation and additional cross-dataset controlled analyses, we find that the effects of SID design are largely non-monotonic: no single SID design is universally best, and commonly used RQ-VAE- and OPQ-based designs can behave inconsistently across datasets. The method with the most balanced first-level codebook is not consistently the best recommender, showing that utilization is diagnostic but insufficient. Scaling either the generative backbone or the SID length is also not always beneficial. Finally, semantic-neighborhood analysis reveals that no single SID design dominates all notions of local semantic preservation; instead, different designs exhibit complementary strengths that remain stable across datasets and neighborhood sizes. Our study provides a controlled and reproducible understanding of semantic ID design and offers practical insights for future generative recommender systems.