Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis
2026-06-22 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how to better teach large language models (LLMs) using fake data by focusing on the right mix of information across topics. They found that simply creating data without considering how knowledge is spread leads to uneven learning. Their method, KDoS, adjusts data creation based on how much knowledge each topic holds, aiming to cover more ground effectively. Tests showed that their approach works well across different models and sizes, improving performance on knowledge tests. This offers a smarter way to make synthetic training data for language models.
Large Language ModelsSynthetic DataKnowledge InjectionKnowledge DistributionData SynthesisKnowledge BoundaryFeedback MechanismWikipedia DatasetModel ScaleKnowledge Benchmark
Authors
Songze Li, Yarong Lan, Zhongpu Bo, Zhaoyang Wang, Zhiqiang Liu, Yuan Yuan, Chengtao Gan, Menghao Qian, Enpei Niu, Xiaoke Guo, Yuanxiang Liu, Zhaoyan Gong, Xiangjin Hu, Liangyurui Liu, Jingdian Lu, Lei Liang, Jun Zhou, Huajun Chen, Wen Zhang
Abstract
Knowledge injection via synthetic data is crucial for enhancing Large Language Models (LLMs). However, current synthesis methods simply stop at preset token counts or fixed data ratios, lacking awareness of knowledge distribution. This results in some domains being sparse while others are redundant, limiting LLM knowledge boundaries. We revisit knowledge injection from a distribution perspective and hypothesize that an optimal knowledge distribution exists to maximize knowledge boundary expansion. We propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism, shifting from blind generation to distribution-optimized synthesis. We construct Wikipedia-based synthetic data with varying knowledge distributions and conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens. Our key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks. Our work offers a new perspective and practical framework for synthetic data-driven knowledge injection.