Towards a Densing Law for User Representation Learning at Billion-Scale Capacity
2026-08-24 • Information Retrieval
Information RetrievalArtificial Intelligence
AI summaryⓘ
The authors study how to better represent user behaviors in large-scale systems by looking at how breaking down data into tokens helps when handling massive amounts of data. They find that simply using raw data hits a performance limit, but tokenizing the data keeps improving results, especially when token settings match data size well. By analyzing this, they discover a pattern linking the needed token capacity to the amount of data, which changes depending on the data type and tokenization method. Using these insights, the authors create ALGN, a new tokenization approach that adapts token length for better results across various tasks, showing their theory works in practice.
user representation learningtokenizationbehavioral sequencescaling lawAlipay datasetvariable-length tokenizationdata redundancydata uniquenesslarge-scale learningadaptive tokenization
Authors
Bin Dou, Junru Zhang, Zhaoyi Yuan, Wuliang Huang, Letian Gong, Baokun Wang, Huan Li, Yu Cheng, Weiqiang Wang
Abstract
User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.