Rethinking Item Tokenization in Generative Recommenders: From Fixed Atoms to Semantic Subwords
2026-08-24 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors found that when recommending items using token sequences, too much focus was put on small details inside each item rather than the bigger patterns between items in a user's history. To fix this, they created a method called Semantic Subword Tokenization (SST), which groups tokenized item parts into meaningful chunks and highlights important transitions between items. This approach helps the system pay more attention to how users behave across items instead of inside them. Their tests showed SST improved recommendation performance compared to previous tokenization methods.
generative recommender systemstokenizationautoregressive predictionuser-context modelingattention mechanismsemantic subwordsbehavioral transitionsencodervariable-length tokensco-occurrence augmentation
Authors
Xinrui Miao, Mingjia Yin, Jiaqing Zhang, Wei Guo, Yong Liu, Yuyang Ye, Hao Wang, Enhong Chen
Abstract
In generative recommender systems, items are typically tokenized into fixed-length semantic ID sequences for autoregressive next-item prediction. However, for user-context modeling, this fine-grained representation triggers Intra-item Attention Overload: excessive attention is spent on low-level intra-item dependencies rather than high-level inter-item behavioral transitions. To address this, we propose Semantic Subword Tokenization (SST), which represents historical items as variable-length semantic subwords while preserving fixed-length target decoding. SST first applies Item-level Subword Tokenization (IST) to merge stable adjacent atom tokens into compact semantic subword tokens, thereby reducing intra-item reassembly in the encoder. It then introduces Behavior-induced Co-occurrence Augmentation (BCA) to inject coarse-grained semantic prefix transition signals, guiding the freed modeling capacity toward inter-item behavioral regularities. Extensive experiments on three public datasets and three generative recommender backbones show empirical improvements of SST over fixed-length and transferable variable-length SID baselines. Code is available at https://github.com/mxrcandy/Semantic-Subword-Tokenization.