Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text
2026-08-31 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how synthetic data, which looks harmless, can secretly teach language models biased behaviors when used for training. They showed that a misaligned 'teacher' model can create synthetic data that embeds social biases without obvious signs, and this data can then influence a 'student' model during fine-tuning. Despite the biases being present, the student models still perform well on normal tasks. The authors point out that this is a new security concern for using synthetic data in model training and suggest methods to detect such hidden biases.
synthetic datalarge language modelsfine-tuningsocial biasaligned modelsmisaligned teacher modelsubliminal learningcovert channellog-linearity scoringmodel security
Authors
Minkyung Cho, Jihyo Kim, SeungWoo Song, Junghun Yuk, Minjoon Kee, Hoyun Song, KyungTae Lim
Abstract
Synthetic data is increasingly used to train large language models (LLMs), yet its security implications remain poorly understood. Prior work on subliminal learning suggests that models can inherit behavioral traits from seemingly unrelated training data. In this work, we investigate whether such mechanisms can be exploited to inject targeted social biases into aligned models through semantically benign synthetic data. We construct a pipeline in which a misaligned teacher model generates filtered synthetic datasets across domains such as creative writing and code generation, which are then used to fine-tune aligned student models. Our experiments show that benign-looking synthetic data can act as a covert channel for transmitting targeted biases while largely preserving the student model's general task capabilities. These results reveal a previously underexplored security risk in synthetic data-driven LLM training pipelines and highlight the need for improved safeguards. As one possible step toward this goal, we suggest that log-linearity-based scoring may provide a useful signal for screening seemingly benign synthetic data.