How Reliable Are Semantic-ID Tokenizer Comparisons in Generative Recommendation?
2026-05-25 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors looked at a way to recommend items using sequences of codes called Semantic-IDs (SID). They found that sometimes different items get the same SID code because the codes group similar items together, which means the usual way of measuring recommendation success can be misleading. Specifically, their tests showed that up to 30% of items share codes, causing big overestimates in recommendation accuracy. To fix this, the authors created new ways to measure performance that account for these code collisions and a method to reduce them, making evaluations more accurate.
Semantic-IDgenerative recommendationautoregressive modeltokenizercode collisiontop-K performanceHit@10item-level metrics
Authors
Qian Zhang, Lech Szymanski, Haibo Zhang, Jeremiah D. Deng
Abstract
In Semantic-ID (SID) based generative recommendation, each item is represented as a sequence of discrete codes, and an autoregressive model is trained to generate the SID sequence of the next item; top-K performance is then measured by checking whether the SID sequence of the target item appears among the generated sequences. This evaluation protocol equates SID-level matching with item-level recommendation, an equivalence that holds only when every SID sequence maps to a single item. We show this assumption breaks down in practice: because tokenizers compress item features into a code space, semantically similar but collaboratively distinct items are frequently assigned the same SID sequence. Across four datasets and five representative tokenizers, the fraction of items involved in such collisions reaches 30.5%, so matching a shared SID sequence identifies only a collision group rather than the target item. Consequently, SID-level metrics overestimate item-level performance (Hit@10 is inflated by up to 103.36%), and the inflation grows with the collision rate. To support faithful comparison, we develop collision-aware item-level metrics computed directly from generated SID sequences, together with a post-tokenizer procedure that reassigns last-level SIDs at minimum cost to obtain a collision-free assignment for any existing tokenizer. Our results indicate that SID-level rankings in prior work should be interpreted with caution, and that reliable tokenizer evaluation requires either item-level correction or collision-free SID assignments.