Optimized semantic tokenization improves large model based recommendations

OptST: Optimizing Semantic Tokenization for LLM-Enhanced Recommendation

Information Retrieval

Summary

Recommender systems suggest things like products or movies by understanding user profiles. This paper discusses a method called OptST that improves the way user information from large language models (LLMs) is turned into tokens for recommendations. Instead of fixing these tokens before training the recommendation system, OptST lets the system adjust them during training to better suit the recommendation task. This method helps the recommender understand users more accurately and performs better in tests on public datasets.

What this means in practice

  • For e-commerce platform teams: Enhance product recommendations by dynamically refining user profile tokens derived from large language models during training to improve prediction accuracy.$Commercial implications: This enables more accurate recommendation engines, which can be offered to online retailers as a competitive product feature.
  • For streaming service developers: Improve content recommendations by adapting semantic tokenization of viewer profiles in real time to better reflect user preferences.

Authors

Hanchong Chen, Jie Liu, Fuyuan Lyu, Dugang Liu, Lingjie Li, Xiuqiang He, Xing Tang

Abstract

Semantic information can enrich the identifiers and structured fields used by recommender systems. Semantic tokenization provides a structured representation of this information by converting LLM-derived user profiles into discrete tokens. In the common two-stage pipeline, the tokenizer is trained before the recommender. Its token assignments are then treated as fixed input features during recommendation training. As a result, the downstream recommender can learn how to use the assigned indices, but it cannot revise the semantic partition that produced them. Making this partition task-aware is difficult because nearest-code assignment is discrete, and unconstrained adaptation may also distort the geometry learned during pretraining. Moreover, a fixed token depth exposes the same semantic resolution to every interaction. We thus propose OptST (Optimizing Semantic Tokenization), which makes a pretrained tokenizer responsive to downstream recommendation. To revise token assignments without directly updating the pretrained encoder or base codebooks, OptST learns a low-rank deformation whose drift is limited by an anchor regularizer. Recommendation supervision reaches this transformation through a straight-through estimator, while prediction retains hard nearest-code assignments. Beyond adapting the assignments themselves, OptST selects an interaction-conditioned prefix to control how much residual detail is exposed to the recommender. Experiments on four public datasets show that OptST achieves the highest AUC among conventional and LLM-enhanced baselines. Its improvements also persist across different recommendation backbones. These results establish task-aware semantic tokenization as an effective interface between LLM-derived profiles and downstream recommendation models.