Choosing near-duplicate images transfers preferences to vision-language models
Near-Duplicate Image Selection Transfers Preferences in Vision-Language Models
Machine Learning
Summary
This paper shows that picking nearly identical versions of the same images can influence what an AI model learns to prefer, even when the pictures and captions don’t change. The authors trained models that selected certain image variants and found that later models trained on those selections inherited these subtle preferences. This effect happens without explicitly telling the models what to prefer and can even be passed on through multiple training rounds. So, even small changes in data selection can quietly shape model behaviors.
What this means in practice
- •For machine learning engineers: Control and influence model training by selecting specific near-duplicate images without changing captions or original photos.
- •For data platform teams: Improve data curation practices by understanding how subtle image variant choices affect downstream vision-language model behavior.
Authors
Jaeyoung Kim, Eunseok Kim, Dongsuk Jang, Seunghyun Kim
Abstract
Subliminal learning is the phenomenon in which a model transmits its preferences to another model through data that do not explicitly express them. As models increasingly curate training data for other models, we ask whether this transfer can arise from choosing among near-duplicate images while keeping the source photographs and human-written captions fixed. For each photograph, a curator selects one of eight near-duplicate variants by caption likelihood, without target-specific prompts. Students trained on target-preferring curators' selections show higher mean target-versus-alternatives log-likelihood ratios in an image-free readout than students trained on neutral selections. Under a prespecified decision rule, this directional shift appears for an owl target in LLaVA-1.5, Qwen2.5-VL, Qwen3-VL, and Gemma 4, with curators and students initialized from the same base checkpoint, and target rotations move it toward camels, oak trees, and a political figure. The owl target-selected student also scores above all count-matched random-reselection students in three models and above 55 of 60 in LLaVA-1.5, and Qwen3-VL shows the shift again on a new source bank with newly trained curators. Its size depends on dataset size, learning rate, and training budget; greedy choices among the eight options move toward the target by 10.4 percentage points in Qwen3-VL and 0.5-1.3 points elsewhere. First-generation students also pass the owl preference to a second generation through captions without explicit owl mentions in LLaVA-1.5 and Qwen3-VL. Thus even a deduplication-style choice among near-identical copies can shift students' target-relative readouts with source photographs and captions fixed; the tested linear probes classify the selecting arm with accuracies up to 59.7% (chance 50%).