Semantic identifier learning improves recommendation at scale
FineSID: Scalable and Efficient Semantic Identifier Learning for Generative Recommendation
Artificial Intelligence
Summary
Many recommendation systems need to represent items with special codes called semantic identifiers to suggest items users might like. Existing methods usually pick only one code per item to learn from, which causes learning problems and poor results. The authors propose FineSID, a new way to spread learning signals softly across many codes at once, leading to better training and fewer conflicts. This approach works well on big sets of items and improves recommendation accuracy without complicated setup.
What this means in practice
- •For e-commerce engineers: Improve personalized product suggestions by using a more effective semantic identifier learning method that handles large item catalogs with fewer errors.
- •For streaming service developers: Enhance content recommendation systems by adopting FineSID for better handling of large content libraries and improving user engagement.
Authors
Song-Li Wu, Weinan Gan, Zhaocheng Du, Xianquan Wang, Jingyi Wang
Abstract
A critical prerequisite of generative recommendation is designing semantic identifiers (SIDs) that are both scalable to large item sets and efficiently learnable. Existing SID learning methods fundamentally rely on Top-1 hard assignment during vector quantization. While heuristic strategies -- such as clustering-based initialization or forced post-hoc collision resolution -- can artificially inflate codebook coverage, they often disrupt end-to-end semantic alignment and fail to address the underlying optimization bottleneck: sparse gradient propagation. In standard Top-1 assignment, gradients concentrate on a narrow subset of frequently selected codewords, leaving the majority inherently under-trained and causing severe SID collisions. To overcome this limitation natively without relying on complex initialization priors, we propose FineSID, a unified quantization framework that moves beyond Top-1 assignment by enabling fine-grained gradient propagation across the entire codebook. Instead of updating only a single selected codeword, FineSID distributes learning signals to all codewords in a soft, differentiable manner. This design promotes globally balanced codebook optimization while strictly preserving semantic consistency, effectively alleviating SID collisions and stabilizing training in large, high-dimensional codebooks. Extensive experiments on multiple public benchmarks demonstrate that FineSID is robust to initialization configurations and consistently improves both codebook utilization and recommendation accuracy. Our work provides a principled, initialization-agnostic solution for semantic identifier learning, advancing the practicality of generative recommendation.