Subliminal learning effects vary in open-weight AI language models
Reproducing and Evaluating the Generalizability of Subliminal Learning in Open-Weight Models
Artificial Intelligence
Summary
Some AI language models can pick up hidden preferences from other models even when trained on unrelated data, a process called subliminal learning. This paper repeats previous experiments and finds that while some hidden traits do transfer, the strength of this effect changes depending on the task, trait, and model used. One tested model barely showed any of this transfer. The study helps clarify when and how subliminal learning happens in openly accessible AI models.
What this means in practice
- •For ai developers: Design training workflows with awareness that some model preferences may transfer unexpectedly depending on traits and tasks.
- •For machine learning engineers: Choose models and data carefully during distillation to minimize undesirable trait inheritance.
Authors
Daan van der Weijden, Nathan Brack, Selene Baez Santamaria
Abstract
In this reproduction paper we investigate subliminal learning, a consequence of distillation where teacher models transmit behavioral preference traits through semantically unrelated data. The original paper explores two types of traits (animal preferences and misalignment), three data modalities (number sequences, code, and chain of thought), and several model families. We reproduce their experiments and extend the setup along three axes: new preference categories (actors and politicians), a new task (chess move generation), and an additional open-weight model (Ministral8B). We also run a controlled ablation on the numbers task's answer-space size (1-, 2-, and 3-digit sequences). We focus on open-weight models with accessible checkpoints on HuggingFace, since the original paper's GPT-4.x fine-tuning is no longer available. Our reproduction supports the original paper's claims, but our extensions show they are not universal as transmission strength varies across traits and tasks, and one model shows almost no effect at all.