Taiji: Pareto Optimal Policy Optimization with Semantics-IDs Trade-off for Industrial LLM-Enhanced Recommendation
2026-06-02 • Information Retrieval
Information RetrievalArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors address challenges in using large language models (LLMs) to improve recommender systems, focusing on how to better align the LLM's understanding with user preferences. They created Taiji, a system that improves training by generating high-quality reasoning data and balances different reward types during fine-tuning. Their new method, POPO, helps find the best trade-off between the LLM's general knowledge and specific user behavior signals. Tests showed Taiji works well, and it has been successfully deployed on a large platform serving hundreds of millions daily.
Large Language ModelsRecommender SystemsSupervised Fine-Tuning (SFT)Reinforcement Learning (RL)Chain-of-Thought (CoT)Pareto Optimal Policy Optimization (POPO)Semantic SpaceCollaborative FilteringReward BalancingUser Preference Modeling
Authors
Yuecheng Li, Zeyu Song, Jing Yao, Chi Lu, Peng Jiang, Kun Gai
Abstract
Scaling recommender systems via large language models (LLMs) has become a prominent trend in the industry. However, aligning the LLM's semantic space with the recommender's ID space via post-training (e.g., SFT and RL) remains challenging. Existing LLM4Rec paradigms are bottlenecked by two main issues: (1) the difficulty of measuring and improving chain-of-thought (CoT) quality in open-domain recommendation during SFT, and (2) the neglect of the trade-off between LLM semantic rewards and recommendation preference rewards during RL alignment. Inspired by these challenges, we present Taiji, a novel LLM-as-Enhancer framework designed for industrial recommender systems. To overcome the SFT bottleneck, we utilize reverse-engineered reasoning and open-ended rejection sampling to generate high-quality, domain-specific CoT data. To resolve the RL alignment issue, we propose Pareto Optimal Policy Optimization (POPO), which adaptively adjusts cross-domain reward weights. Theoretically, it achieves an optimal trade-off between the semantic world knowledge of LLMs and the collaborative ID features representing online user preferences. Extensive offline evaluations and online A/B tests validate the effectiveness of Taiji. Deployed on Kuaishou's advertising platform since May 2026, Taiji currently serves over 400 million users daily, yielding significant commercial revenue and demonstrating its robust scalability in web-scale environments.